[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:24 dist-test-slave-6kf3 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:24 dist-test-slave-6kf3 krb5kdc[399](info): set up 2 sockets
Sep 24 19:36:24 dist-test-slave-6kf3 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 19:36:25.599648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:25 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:25.710367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:25.788565 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:25.788800 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:25.788833 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:25.790009 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:25.790056 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:25.790084 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:25.790099 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:25.790114 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:25.791651 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:25.791909 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:25.792171 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:25.797412 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:25.797464 412 server_base.cc:1034] running on GCE node
W20260924 19:36:25.797421 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:25.797451 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:25.797801 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:25.798075 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:25.799216 412 hybrid_clock.cc:648] HybridClock initialized: now 1790278585799190 us; error 40 us; skew 500 ppm
Sep 24 19:36:25 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:25.801623 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:25.802084 412 webserver.cc:460] Webserver started at http://127.0.98.62:34029/ using document root <none> and password file <none>
I20260924 19:36:25.802294 412 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:25.802340 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:25.802465 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:25.803536 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "bb885aeb69254580bcf1aca961894239"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "846a002f2d44b3be6f8e834a152c7cb7"
server_key_iv: "eb52580c604fba6eaa95965d4365ccae"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.803905 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb885aeb69254580bcf1aca961894239"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "846a002f2d44b3be6f8e834a152c7cb7"
server_key_iv: "eb52580c604fba6eaa95965d4365ccae"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.805195 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:25.805814 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:25.805948 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:25.806046 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "bb885aeb69254580bcf1aca961894239"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "846a002f2d44b3be6f8e834a152c7cb7"
server_key_iv: "eb52580c604fba6eaa95965d4365ccae"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.806103 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:25.822531 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:25.824556 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:25.824712 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:25.828457 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44657
I20260924 19:36:25.828498 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44657 every 8 connection(s)
I20260924 19:36:25.828837 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:25.829674 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:25.831840 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239: Bootstrap starting.
I20260924 19:36:25.832410 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:25.832687 480 log.cc:824] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:25.833451 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239: No bootstrap required, opened a new log
I20260924 19:36:25.834669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 412
I20260924 19:36:25.834753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:25.835103 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 19:36:25.835240 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:25.835260 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb885aeb69254580bcf1aca961894239, State: Initialized, Role: FOLLOWER
I20260924 19:36:25.835386 392 external_mini_cluster.cc:1428] Setting key ae402a05076e999445a4a9603f06569d
I20260924 19:36:25.835388 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 19:36:25.835449 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:25.835479 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:25.835536 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:25.836084 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 19:36:25.836181 480 leader_election.cc:302] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb885aeb69254580bcf1aca961894239; no voters:
I20260924 19:36:25.836313 480 leader_election.cc:288] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:25.836575 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:25.836673 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:25.836773 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [term 1 LEADER]: Becoming Leader. State: Replica: bb885aeb69254580bcf1aca961894239, State: Running, Role: LEADER
I20260924 19:36:25.836894 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 19:36:25.837208 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb885aeb69254580bcf1aca961894239. Latest consensus state: current_term: 1 leader_uuid: "bb885aeb69254580bcf1aca961894239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } } }
I20260924 19:36:25.837335 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:25.837349 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb885aeb69254580bcf1aca961894239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb885aeb69254580bcf1aca961894239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } } }
I20260924 19:36:25.837430 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:25.837594 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:25.838131 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:25.840376 494 catalog_manager.cc:1347] Generated new cluster ID: 3e7658f0f12440ac9536221884af9721
I20260924 19:36:25.840425 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:25.844343 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:25.844973 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:25.848208 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb885aeb69254580bcf1aca961894239: Generated new TSK 0
I20260924 19:36:25.848361 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 24 19:36:25 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:25.853012 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:25.836191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.0.98.62:44657)
0924 19:36:25.836372 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:25.836382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:25.836667 (+ 285us) server_negotiation.cc:407] Connection header received
0924 19:36:25.839016 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:25.839024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:25.839080 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:25.839185 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:25.841402 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:25.842763 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:25.844313 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:25.844893 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:25.844996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:25.849083 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:25.849088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:25.849089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:25.849101 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:25.851082 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:25.851507 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:25.851510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:25.851510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:25.851625 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:25.851897 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:25.851898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:25.851898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:25.852231 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:25.852359 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:25.852613 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:36:25.852765 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:25.879236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:25.963887 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:25.964094 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:25.964116 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:25.965245 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:25.965286 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:25.965310 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:25.966749 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:25.966955 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:25.967170 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:25.971836 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:25.971905 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:25.971861 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:25.972093 506 server_base.cc:1034] running on GCE node
I20260924 19:36:25.972262 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:25.972486 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:25.973647 506 hybrid_clock.cc:648] HybridClock initialized: now 1790278585973635 us; error 25 us; skew 500 ppm
Sep 24 19:36:25 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:25.976017 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:25.976431 506 webserver.cc:460] Webserver started at http://127.0.98.1:41535/ using document root <none> and password file <none>
I20260924 19:36:25.976684 506 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:25.976745 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:25.976873 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:25.978253 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "d95512afa5b846a18233a90bdb403df2"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "70f0fb878dcbc48db948b6c9f72731c7"
server_key_iv: "1b4caa63cdf72c0ac143341ad240f02c"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.978703 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d95512afa5b846a18233a90bdb403df2"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "70f0fb878dcbc48db948b6c9f72731c7"
server_key_iv: "1b4caa63cdf72c0ac143341ad240f02c"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.980083 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:25.980777 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:25.980916 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:25.980994 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "d95512afa5b846a18233a90bdb403df2"
format_stamp: "Formatted at 2026-09-24 19:36:25 on dist-test-slave-6kf3"
server_key: "70f0fb878dcbc48db948b6c9f72731c7"
server_key_iv: "1b4caa63cdf72c0ac143341ad240f02c"
server_key_version: "encryptionkey@0"
I20260924 19:36:25.981051 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:25.988802 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:25.990777 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:25.990903 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:25.991108 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:25.991372 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:25.991402 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:25.991430 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:25.991453 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:25.997123 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38885
I20260924 19:36:25.997172 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38885 every 8 connection(s)
I20260924 19:36:25.997463 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:36:25 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:26.004122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 506
I20260924 19:36:26.004204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:26.004341 392 external_mini_cluster.cc:1428] Setting key 5adad1ada7e1eea793629ce3dd0d1bed
I20260924 19:36:26.006646 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:25.998123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60051 (local address 127.0.98.62:44657)
0924 19:36:25.998167 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:25.998169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:25.998357 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:36:25.998814 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:25.998815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:25.998824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:25.998870 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:25.999813 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.000401 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.001405 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.001617 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.001677 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.003673 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:26.003675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:26.003675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:26.003682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:26.005119 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.005572 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.005573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.005573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.005638 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.005904 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.005906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.005906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.006029 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:26.006130 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:26.006436 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:36:26.006550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:26.006708 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:25.998103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44657 (local address 127.0.98.1:60051)
0924 19:36:25.998243 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:25.998278 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:25.998739 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:25.998934 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:25.998937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:25.999515 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:25.999711 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:25.999716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.000472 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.000473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.001314 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:26.001316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.001686 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.001687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.001782 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.002277 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:26.002301 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:26.003577 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:26.005192 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.005199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.005202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.005477 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.005696 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.005697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.005697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.005826 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.006123 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:26.006129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:26.006309 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 19:36:26.006398 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:36:26.006528 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 19:36:26.007622 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44657
I20260924 19:36:26.007716 635 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:26.007898 635 heartbeater.cc:511] Master 127.0.98.62:44657 requested a full tablet report, sending...
I20260924 19:36:26.008437 444 ts_manager.cc:194] Registered new tserver with Master: d95512afa5b846a18233a90bdb403df2 (127.0.98.1:38885)
I20260924 19:36:26.009085 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60051
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:26.028668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:26.105899 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:26.106101 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:26.106130 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:26.107225 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:26.107265 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:26.107299 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:26.108747 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:26.108966 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:26.109166 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:26.113729 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:26.113752 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:26.113780 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:26.113866 640 server_base.cc:1034] running on GCE node
I20260924 19:36:26.114101 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:26.114323 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:26.115458 640 hybrid_clock.cc:648] HybridClock initialized: now 1790278586115445 us; error 26 us; skew 500 ppm
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:26.118006 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:26.118415 640 webserver.cc:460] Webserver started at http://127.0.98.2:42649/ using document root <none> and password file <none>
I20260924 19:36:26.118605 640 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:26.118645 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:26.118734 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:26.119728 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "c864e2506117457d94aa42b766b994a9"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "d15235e20abc43651c189139f14f1912"
server_key_iv: "930e64311da5edfda9aebeeb02f486f9"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.120081 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c864e2506117457d94aa42b766b994a9"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "d15235e20abc43651c189139f14f1912"
server_key_iv: "930e64311da5edfda9aebeeb02f486f9"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.121335 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:26.121987 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.122133 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:26.122224 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "c864e2506117457d94aa42b766b994a9"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "d15235e20abc43651c189139f14f1912"
server_key_iv: "930e64311da5edfda9aebeeb02f486f9"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.122284 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:26.136891 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:26.139174 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:26.139300 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:26.139531 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:26.139840 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:26.139876 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.139921 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:26.139978 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.146134 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39137
I20260924 19:36:26.146178 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39137 every 8 connection(s)
I20260924 19:36:26.146484 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:26.153124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 640
I20260924 19:36:26.153220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:26.153393 392 external_mini_cluster.cc:1428] Setting key fb781fc82096694f3632bb13db653338
I20260924 19:36:26.155723 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.147191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36715 (local address 127.0.98.62:44657)
0924 19:36:26.147314 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:26.147316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:26.147491 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:36:26.148024 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:26.148026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:26.148037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:26.148093 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:26.149044 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.149535 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.150523 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.150708 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.150761 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.152837 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:26.152839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:26.152839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:26.152846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:26.154468 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.154815 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.154816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.154817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.154878 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.155087 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.155087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.155088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.155214 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:26.155285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:26.155539 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:36:26.155635 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 19:36:26.155797 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.147149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44657 (local address 127.0.98.2:36715)
0924 19:36:26.147342 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:26.147384 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:26.147883 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:26.148177 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:26.148181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:26.148743 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:26.148946 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:26.148951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.149615 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.149616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.150411 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:26.150413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.150772 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.150773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.150889 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.151372 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:26.151380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:26.152756 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:26.154538 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.154543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.154545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.154751 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.154945 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.154946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.154946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.155038 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.155269 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:26.155273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:26.155390 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 19:36:26.155491 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:36:26.155611 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260924 19:36:26.156522 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44657
I20260924 19:36:26.156622 769 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:26.156811 769 heartbeater.cc:511] Master 127.0.98.62:44657 requested a full tablet report, sending...
I20260924 19:36:26.157130 444 ts_manager.cc:194] Registered new tserver with Master: c864e2506117457d94aa42b766b994a9 (127.0.98.2:39137)
I20260924 19:36:26.157498 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36715
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:26.178275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:26.257411 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:26.257606 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:26.257637 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:26.258754 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:26.258792 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:26.258826 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:26.260243 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:26.260469 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:26.260676 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:26.265954 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:26.265964 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:26.265962 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:26.266070 774 server_base.cc:1034] running on GCE node
I20260924 19:36:26.266311 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:26.266546 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:26.267676 774 hybrid_clock.cc:648] HybridClock initialized: now 1790278586267664 us; error 28 us; skew 500 ppm
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:26.270040 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:26.270427 774 webserver.cc:460] Webserver started at http://127.0.98.3:34879/ using document root <none> and password file <none>
I20260924 19:36:26.270638 774 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:26.270679 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:26.270793 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:26.271766 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d201d81d2ff4561a98a45146385efa4"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "77ad0320a7015d3ee9b9d9d4a2c20d1f"
server_key_iv: "6417461f12d3dc1a3feeeedea768365d"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.272137 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d201d81d2ff4561a98a45146385efa4"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "77ad0320a7015d3ee9b9d9d4a2c20d1f"
server_key_iv: "6417461f12d3dc1a3feeeedea768365d"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.273309 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:26.273926 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.274070 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:26.274142 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "3d201d81d2ff4561a98a45146385efa4"
format_stamp: "Formatted at 2026-09-24 19:36:26 on dist-test-slave-6kf3"
server_key: "77ad0320a7015d3ee9b9d9d4a2c20d1f"
server_key_iv: "6417461f12d3dc1a3feeeedea768365d"
server_key_version: "encryptionkey@0"
I20260924 19:36:26.274205 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:26.289451 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:26.291201 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:26.291325 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:26.291549 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:26.291839 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:26.291870 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.291893 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:26.291915 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:26.297839 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44389
I20260924 19:36:26.297919 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44389 every 8 connection(s)
I20260924 19:36:26.298218 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:26.302623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 774
I20260924 19:36:26.302706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:26.302857 392 external_mini_cluster.cc:1428] Setting key 5d87290a8d2b7714c393f3fe88e82735
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:26.307044 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.298924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56691 (local address 127.0.98.62:44657)
0924 19:36:26.298975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:26.298977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:26.299532 (+ 555us) server_negotiation.cc:407] Connection header received
0924 19:36:26.300003 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:26.300005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:26.300016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:26.300087 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:26.300953 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.301475 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.302454 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.302647 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.302722 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.304994 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:26.304995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:26.304995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:26.305001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:26.306068 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.306383 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.306383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.306383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.306425 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.306586 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.306586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.306587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.306679 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:26.306723 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:26.306925 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:36:26.306978 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:36:26.307214 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.298886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44657 (local address 127.0.98.3:56691)
0924 19:36:26.299033 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:26.299074 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:26.299902 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:26.300143 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:26.300146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:26.300676 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:26.300873 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:26.300877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.301562 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.301564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.302368 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:26.302371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.302737 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.302739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:26.302881 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.303479 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:26.303488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:26.304911 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:26.306127 (+ 1216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.306132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.306135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.306330 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.306460 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:26.306460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:26.306460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:26.306547 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:26.306721 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:26.306723 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:26.306830 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:36:26.306902 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:36:26.307016 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260924 19:36:26.307619 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44657
I20260924 19:36:26.307704 903 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:26.307868 903 heartbeater.cc:511] Master 127.0.98.62:44657 requested a full tablet report, sending...
I20260924 19:36:26.308220 444 ts_manager.cc:194] Registered new tserver with Master: 3d201d81d2ff4561a98a45146385efa4 (127.0.98.3:44389)
I20260924 19:36:26.308627 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56691
I20260924 19:36:26.314764 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:26.335820 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.329233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.98.1:38885)
0924 19:36:26.329388 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:26.329393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:26.329402 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:26.329427 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:26.329429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:26.329442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:26.329502 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:26.329999 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.330688 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.331353 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.331652 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.331704 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.333727 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:26.333735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:26.333735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:26.333742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:26.334977 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.335243 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.335245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.335246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.335307 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.335423 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.335424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.335424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.335552 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:26.335654 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:26.335687 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 19:36:26.335744 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260924 19:36:26.336205 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37558
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:26.355736 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:26.349705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.0.98.1:38885)
0924 19:36:26.349767 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:26.349769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:26.349808 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:36:26.349881 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:26.349882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:26.349895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:26.349939 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:26.350436 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.350916 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.351633 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:26.351839 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:26.351893 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:26.353817 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:26.353818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:26.353818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:26.353828 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:26.354824 (+ 996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.355099 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.355100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.355101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.355148 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:26.355299 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:26.355300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:26.355300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:26.355387 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:26.355487 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:26.355566 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:36:26.355643 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:36:26.356225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 506
I20260924 19:36:26.360765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 640
I20260924 19:36:26.365378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 774
I20260924 19:36:26.368993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 412
2026-09-24T19:36:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1725 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[917](info): set up 2 sockets
Sep 24 19:36:26 dist-test-slave-6kf3 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 19:36:27.311200 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:27.416909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:27.494706 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:27.494904 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:27.494925 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:27.496066 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:27.496107 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:27.496120 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:27.496131 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:27.496155 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:27.497912 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:27.498142 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:27.498359 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:27.502815 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.502839 930 server_base.cc:1034] running on GCE node
W20260924 19:36:27.502846 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.502822 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.503151 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:27.503415 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:27.504539 930 hybrid_clock.cc:648] HybridClock initialized: now 1790278587504525 us; error 31 us; skew 500 ppm
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:27.506702 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:27.507104 930 webserver.cc:460] Webserver started at http://127.0.98.62:44737/ using document root <none> and password file <none>
I20260924 19:36:27.507328 930 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:27.507375 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:27.507498 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:27.508543 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "647792679e724094b53e7b1c4196f376"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "d3a410b2c80a998ce1392903dd837fa1"
server_key_iv: "3d3d722e48df3af35f60f83387ef2ffe"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.508859 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "647792679e724094b53e7b1c4196f376"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "d3a410b2c80a998ce1392903dd837fa1"
server_key_iv: "3d3d722e48df3af35f60f83387ef2ffe"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.509999 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:27.510609 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.510758 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:27.510826 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "647792679e724094b53e7b1c4196f376"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "d3a410b2c80a998ce1392903dd837fa1"
server_key_iv: "3d3d722e48df3af35f60f83387ef2ffe"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.510883 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:27.518946 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:27.520596 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:27.520717 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:27.524263 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34407
I20260924 19:36:27.524296 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34407 every 8 connection(s)
I20260924 19:36:27.524596 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:27.525339 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:27.527462 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: Bootstrap starting.
I20260924 19:36:27.527993 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:27.528227 998 log.cc:824] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:27.528792 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: No bootstrap required, opened a new log
I20260924 19:36:27.530292 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } }
I20260924 19:36:27.530421 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:27.530441 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647792679e724094b53e7b1c4196f376, State: Initialized, Role: FOLLOWER
I20260924 19:36:27.530539 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } }
I20260924 19:36:27.530588 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:27.530606 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:27.530638 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:27.530858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 930
I20260924 19:36:27.530936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:27.531085 392 external_mini_cluster.cc:1428] Setting key f98e3a98e220b3a6cb130329f7a9558b
I20260924 19:36:27.531239 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } }
I20260924 19:36:27.531337 998 leader_election.cc:302] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647792679e724094b53e7b1c4196f376; no voters:
I20260924 19:36:27.531479 998 leader_election.cc:288] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:27.531729 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:27.531786 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:27.531916 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [term 1 LEADER]: Becoming Leader. State: Replica: 647792679e724094b53e7b1c4196f376, State: Running, Role: LEADER
I20260924 19:36:27.532035 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } }
I20260924 19:36:27.532387 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647792679e724094b53e7b1c4196f376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } } }
I20260924 19:36:27.532460 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:27.532589 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647792679e724094b53e7b1c4196f376. Latest consensus state: current_term: 1 leader_uuid: "647792679e724094b53e7b1c4196f376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647792679e724094b53e7b1c4196f376" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34407 } } }
I20260924 19:36:27.532642 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376 [sys.catalog]: This master's current role is: LEADER
W20260924 19:36:27.533454 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:27.533505 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:27.533556 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:27.533728 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:27.535006 1012 catalog_manager.cc:1347] Generated new cluster ID: 3249fb2c979e4b7ea669a7b24c791d72
I20260924 19:36:27.535053 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:27.542598 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:27.531529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.0.98.62:34407)
0924 19:36:27.531954 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:27.531963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:27.531995 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:36:27.532108 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:27.532116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:27.532177 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:27.533081 (+ 904us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:27.533302 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.534336 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.535321 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.535914 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.536042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:27.538681 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:27.538684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:27.538685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:27.538696 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:27.541096 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.541469 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.541471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.541471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.541562 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.541754 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.541755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.541755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.542030 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:27.542167 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:27.542268 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:36:27.542398 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":344,"thread_start_us":68,"threads_started":1}
I20260924 19:36:27.544059 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:27.544486 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:27.550916 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647792679e724094b53e7b1c4196f376: Generated new TSK 0
I20260924 19:36:27.551079 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:27.574033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:27.651383 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:27.651583 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:27.651614 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:27.652782 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:27.652823 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:27.652858 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:27.654280 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:27.654497 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:27.654699 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:27.659199 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.659240 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.659205 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.659371 1024 server_base.cc:1034] running on GCE node
I20260924 19:36:27.659564 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:27.659775 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:27.660903 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790278587660888 us; error 31 us; skew 500 ppm
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:27.663177 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:27.663576 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36981/ using document root <none> and password file <none>
I20260924 19:36:27.663779 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:27.663848 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:27.663990 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:27.665022 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf6a21d3beac48e7b6fdb181b03a5d89"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "519070e0a8bbcc22294ad2c76c3f3d48"
server_key_iv: "cd0883e1329b10d786c627dc61e53ce8"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.665336 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf6a21d3beac48e7b6fdb181b03a5d89"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "519070e0a8bbcc22294ad2c76c3f3d48"
server_key_iv: "cd0883e1329b10d786c627dc61e53ce8"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.666458 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:27.667074 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.667210 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:27.667289 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "bf6a21d3beac48e7b6fdb181b03a5d89"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "519070e0a8bbcc22294ad2c76c3f3d48"
server_key_iv: "cd0883e1329b10d786c627dc61e53ce8"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.667338 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:27.681576 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:27.683269 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:27.683393 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:27.683580 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:27.683846 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:27.683876 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.683897 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:27.683912 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.689241 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38837
I20260924 19:36:27.689308 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38837 every 8 connection(s)
I20260924 19:36:27.689576 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:27.697543 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:27.690210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52619 (local address 127.0.98.62:34407)
0924 19:36:27.690287 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:27.690289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:27.690469 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:36:27.690938 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:27.690939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:27.690947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:27.690994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:27.691875 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.692406 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.693336 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.693518 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.693562 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:27.695470 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:27.695471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:27.695471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:27.695477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:27.696585 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.696905 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.696905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.696906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.696947 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.697104 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.697105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.697105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.697192 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:27.697234 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:27.697428 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 19:36:27.697480 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:36:27.697745 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:27.690205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34407 (local address 127.0.98.1:52619)
0924 19:36:27.690355 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:27.690391 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:27.690867 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:27.691051 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:27.691054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:27.691591 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:27.691790 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:27.691795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.692478 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.692478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:27.693276 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:27.693278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.693571 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.693571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:27.693666 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:27.694147 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:27.694155 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:27.695394 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:27.696645 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:27.696651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:27.696652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:27.696852 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:27.696986 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:27.696986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:27.696987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:27.697067 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:27.697231 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:27.697233 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:27.697339 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:36:27.697411 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:36:27.697520 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260924 19:36:27.698194 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34407
I20260924 19:36:27.698273 1153 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:27.698410 1153 heartbeater.cc:511] Master 127.0.98.62:34407 requested a full tablet report, sending...
I20260924 19:36:27.698632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1024
I20260924 19:36:27.698716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:27.698875 392 external_mini_cluster.cc:1428] Setting key 7bba5aca8291e6080360f8ed46151762
I20260924 19:36:27.698909 962 ts_manager.cc:194] Registered new tserver with Master: bf6a21d3beac48e7b6fdb181b03a5d89 (127.0.98.1:38837)
I20260924 19:36:27.699519 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52619
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:27.720691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:27.798492 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:27.798702 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:27.798731 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:27.799865 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:27.799906 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:27.799968 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:27.801385 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:27.801621 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:27.801829 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:27.806432 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.806492 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.806535 1158 server_base.cc:1034] running on GCE node
W20260924 19:36:27.806442 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.806814 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:27.807027 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:27.808163 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790278587808151 us; error 26 us; skew 500 ppm
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:27.810413 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:27.810827 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34461/ using document root <none> and password file <none>
I20260924 19:36:27.811030 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:27.811076 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:27.811203 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:27.812249 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "206c615b5d7a481aaa8b7b486afed2ee"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "dd92ac8fc5f789898c99e10e7b2ff3a7"
server_key_iv: "64fe7569332bb8f0618eb7202ef52428"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.812553 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "206c615b5d7a481aaa8b7b486afed2ee"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "dd92ac8fc5f789898c99e10e7b2ff3a7"
server_key_iv: "64fe7569332bb8f0618eb7202ef52428"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.813732 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:27.814373 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.814514 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:27.814592 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "206c615b5d7a481aaa8b7b486afed2ee"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "dd92ac8fc5f789898c99e10e7b2ff3a7"
server_key_iv: "64fe7569332bb8f0618eb7202ef52428"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.814675 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:27.834721 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:27.836856 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:27.836992 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:27.837213 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:27.837499 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:27.837531 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.837559 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:27.837579 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.842846 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40963
I20260924 19:36:27.842900 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40963 every 8 connection(s)
I20260924 19:36:27.843214 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:27.844724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1158
I20260924 19:36:27.844815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:27.844974 392 external_mini_cluster.cc:1428] Setting key f7b886a5efdda3a3a6b3cb245105d98d
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:36:27.852560 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:27.843906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55755 (local address 127.0.98.62:34407)
0924 19:36:27.843966 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:27.843968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:27.844161 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:36:27.844683 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:27.844684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:27.844695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:27.844752 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:27.845798 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.846273 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.847383 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.847651 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.847706 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:27.850023 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:27.850024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:27.850025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:27.850031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:27.851239 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.851671 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.851672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.851672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.851715 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:27.851985 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:27.851986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:27.851986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:27.852079 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:27.852125 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:27.852421 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 19:36:27.852496 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:36:27.852778 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:27.843885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34407 (local address 127.0.98.2:55755)
0924 19:36:27.844048 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:27.844084 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:27.844574 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:27.844832 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:27.844837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:27.845462 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:27.845716 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:27.845723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.846348 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.846349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:27.847304 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:27.847307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:27.847730 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:27.847731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:27.847867 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:27.848362 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:27.848372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:27.849937 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:27.851311 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:27.851317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:27.851320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:27.851587 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:27.851766 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:27.851767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:27.851768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:27.851890 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:27.852132 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:27.852138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:27.852293 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 19:36:27.852399 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:36:27.852558 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260924 19:36:27.853242 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34407
I20260924 19:36:27.853324 1287 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:27.853466 1287 heartbeater.cc:511] Master 127.0.98.62:34407 requested a full tablet report, sending...
I20260924 19:36:27.853786 962 ts_manager.cc:194] Registered new tserver with Master: 206c615b5d7a481aaa8b7b486afed2ee (127.0.98.2:40963)
I20260924 19:36:27.854203 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:27.867592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:27.945966 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:27.946197 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:27.946220 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:27.947369 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:27.947410 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:27.947436 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:27.948856 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:27.949043 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:27.949230 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:27.953981 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.954008 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:27.954020 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:27.954243 1292 server_base.cc:1034] running on GCE node
I20260924 19:36:27.954447 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:27.954671 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:27.955806 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790278587955793 us; error 24 us; skew 500 ppm
Sep 24 19:36:27 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:27.958119 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:27.958535 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41865/ using document root <none> and password file <none>
I20260924 19:36:27.958734 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:27.958779 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:27.958896 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:27.959892 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "7d2aaeac9c5641c390e498b6a4b69270"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "f146bc8a602d68c9fb56b0fde71a75ff"
server_key_iv: "37b6227a5c39acdea0af937facff56b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.960263 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7d2aaeac9c5641c390e498b6a4b69270"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "f146bc8a602d68c9fb56b0fde71a75ff"
server_key_iv: "37b6227a5c39acdea0af937facff56b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.961445 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:27.962060 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:27.962193 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:27.962270 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "7d2aaeac9c5641c390e498b6a4b69270"
format_stamp: "Formatted at 2026-09-24 19:36:27 on dist-test-slave-6kf3"
server_key: "f146bc8a602d68c9fb56b0fde71a75ff"
server_key_iv: "37b6227a5c39acdea0af937facff56b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:27.962332 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:28.021281 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:28.023480 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:28.023617 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:28.023833 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:28.024142 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:28.024180 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:28.024209 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:28.024226 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:28.029620 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36483
I20260924 19:36:28.029673 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36483 every 8 connection(s)
I20260924 19:36:28.029968 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:28.032445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1292
I20260924 19:36:28.032533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:28.032680 392 external_mini_cluster.cc:1428] Setting key db6c96a04a0742e3d17c9ad7cd305fd5
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:28.038784 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.030638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52031 (local address 127.0.98.62:34407)
0924 19:36:28.030692 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.030693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.030855 (+ 162us) server_negotiation.cc:407] Connection header received
0924 19:36:28.031371 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.031372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.031382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.031419 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:28.032388 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.033040 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.034161 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.034346 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.034401 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.036527 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:28.036528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:28.036528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:28.036534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:28.037734 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.038080 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.038081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.038082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.038133 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.038316 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.038317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.038317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.038412 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:28.038455 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.038653 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.038715 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:36:28.038945 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.030596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34407 (local address 127.0.98.3:52031)
0924 19:36:28.030738 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.030779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.031266 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.031489 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.031493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.032069 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:28.032291 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.032298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.033211 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.033212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.034092 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.034094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.034413 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.034415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.034535 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.035067 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:28.035075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:28.036455 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:28.037807 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:28.037813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:28.037815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:28.038011 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:28.038184 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:28.038184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:28.038185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:28.038275 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:28.038450 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:28.038453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:28.038562 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.038636 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.038749 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260924 19:36:28.039353 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34407
I20260924 19:36:28.039433 1421 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:28.039570 1421 heartbeater.cc:511] Master 127.0.98.62:34407 requested a full tablet report, sending...
I20260924 19:36:28.039868 962 ts_manager.cc:194] Registered new tserver with Master: 7d2aaeac9c5641c390e498b6a4b69270 (127.0.98.3:36483)
I20260924 19:36:28.040230 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52031
I20260924 19:36:28.044315 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:36:28.050736 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.045692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.0.98.62:34407)
0924 19:36:28.045752 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.045753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.045793 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:36:28.045849 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.045850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.045858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.045892 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:28.046354 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.046811 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.047438 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.047607 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.047646 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.048783 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:28.048785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:28.048785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:28.048790 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:28.049802 (+ 1012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.050105 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.050107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.050107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.050163 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.050309 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.050309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.050309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.050441 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:28.050527 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.050590 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.050648 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:36:28.052764 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:28.054333 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:28.060721 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40963 (local address 127.0.0.1:38450)
0924 19:36:28.056950 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.056956 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.056989 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.057144 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.057144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.057147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:28.057217 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.057219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.058157 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.058158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.059442 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.059445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.060372 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.060373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.060448 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.060465 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.060555 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.060611 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":6,"thread_start_us":85,"threads_started":1}
I20260924 19:36:28.060716 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.0.98.2:40963)
0924 19:36:28.056659 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.056663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.057006 (+ 343us) server_negotiation.cc:407] Connection header received
0924 19:36:28.057048 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.057051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.057065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.057136 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:28.057274 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.058061 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.059516 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.060294 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.060363 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.060414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.060593 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.060639 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260924 19:36:28.060922 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.1:38837)
0924 19:36:28.056840 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.056843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.056905 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:36:28.056949 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.056951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.056964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.057045 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:28.057318 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.058380 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.059856 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.060413 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.060481 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.060533 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.060808 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.060849 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260924 19:36:28.060743 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38837 (local address 127.0.0.1:56210)
0924 19:36:28.056839 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.056848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.056895 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.057138 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.057141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.057144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:28.057243 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.057244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.058481 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.058482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.059782 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.059785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.060482 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.060483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.060547 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.060551 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.060606 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.060647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260924 19:36:28.061743 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36483 (local address 127.0.0.1:60834)
0924 19:36:28.056668 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.056711 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.056778 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.057140 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.057141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.057145 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:28.057241 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.057243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.059374 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.059375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.060457 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.060460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.061413 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.061414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.061580 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.061585 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.061639 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.061678 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":23,"thread_start_us":57,"threads_started":1}
I20260924 19:36:28.061785 1221 tablet_service.cc:1467] Processing CreateTablet for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd (DEFAULT_TABLE table=test-table [id=e21bd12fcd794f3c9a55325f2bfce8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:28.061795 1087 tablet_service.cc:1467] Processing CreateTablet for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd (DEFAULT_TABLE table=test-table [id=e21bd12fcd794f3c9a55325f2bfce8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:28.061991 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.056675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.3:36483)
0924 19:36:28.056901 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.056904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.056913 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:28.056947 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.056949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.056964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.057045 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:28.057383 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.059280 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.060535 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.061191 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.061256 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.061311 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.061881 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.061923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260924 19:36:28.062155 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:28.062157 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:28.062572 1355 tablet_service.cc:1467] Processing CreateTablet for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd (DEFAULT_TABLE table=test-table [id=e21bd12fcd794f3c9a55325f2bfce8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:28.062865 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:28.064782 1438 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Bootstrap starting.
I20260924 19:36:28.065320 1438 tablet_bootstrap.cc:654] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:28.065351 1439 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: Bootstrap starting.
I20260924 19:36:28.065761 1440 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: Bootstrap starting.
I20260924 19:36:28.065886 1438 log.cc:824] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:28.066067 1439 tablet_bootstrap.cc:654] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:28.066329 1440 tablet_bootstrap.cc:654] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:28.066488 1439 log.cc:824] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:28.066756 1438 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: No bootstrap required, opened a new log
I20260924 19:36:28.066813 1438 ts_tablet_manager.cc:1397] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:28.066852 1440 log.cc:824] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:28.067227 1439 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: No bootstrap required, opened a new log
I20260924 19:36:28.067325 1439 ts_tablet_manager.cc:1397] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:28.067481 1440 tablet_bootstrap.cc:492] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: No bootstrap required, opened a new log
I20260924 19:36:28.067536 1440 ts_tablet_manager.cc:1397] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:28.068914 1440 raft_consensus.cc:348] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069056 1440 raft_consensus.cc:374] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:28.069094 1440 raft_consensus.cc:729] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2aaeac9c5641c390e498b6a4b69270, State: Initialized, Role: FOLLOWER
I20260924 19:36:28.069252 1440 consensus_queue.cc:260] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069167 1438 raft_consensus.cc:348] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069219 1439 raft_consensus.cc:348] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069336 1438 raft_consensus.cc:374] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:28.069350 1439 raft_consensus.cc:374] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:28.069379 1439 raft_consensus.cc:729] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6a21d3beac48e7b6fdb181b03a5d89, State: Initialized, Role: FOLLOWER
I20260924 19:36:28.069379 1438 raft_consensus.cc:729] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c615b5d7a481aaa8b7b486afed2ee, State: Initialized, Role: FOLLOWER
I20260924 19:36:28.069471 1439 consensus_queue.cc:260] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069492 1438 consensus_queue.cc:260] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.069545 1440 ts_tablet_manager.cc:1428] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:28.069665 1421 heartbeater.cc:503] Master 127.0.98.62:34407 was elected leader, sending a full tablet report...
I20260924 19:36:28.069732 1439 ts_tablet_manager.cc:1428] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:36:28.069837 1153 heartbeater.cc:503] Master 127.0.98.62:34407 was elected leader, sending a full tablet report...
I20260924 19:36:28.069998 1438 ts_tablet_manager.cc:1428] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:28.070106 1287 heartbeater.cc:503] Master 127.0.98.62:34407 was elected leader, sending a full tablet report...
I20260924 19:36:28.087158 1444 raft_consensus.cc:482] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:28.087273 1444 raft_consensus.cc:504] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.087621 1444 leader_election.cc:288] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 206c615b5d7a481aaa8b7b486afed2ee (127.0.98.2:40963), bf6a21d3beac48e7b6fdb181b03a5d89 (127.0.98.1:38837)
I20260924 19:36:28.090173 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.087905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40265 (local address 127.0.98.1:38837)
0924 19:36:28.087971 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.087973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.087980 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:28.088003 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.088004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.088014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.088057 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:28.088189 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.088656 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.089511 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.089888 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.089944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.089953 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.090072 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.090114 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:36:28.090171 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.087780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38837 (local address 127.0.98.3:40265)
0924 19:36:28.087823 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.087836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.087878 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.088070 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.088071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.088073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:28.088135 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.088137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.088721 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.088722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.089434 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.089436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.089950 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.089951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.090018 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.090021 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.090062 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.090101 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:36:28.090448 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7f0fa06c5f45b2aabebbed15a2a7dd" candidate_uuid: "7d2aaeac9c5641c390e498b6a4b69270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" is_pre_election: true
I20260924 19:36:28.090600 1107 raft_consensus.cc:2454] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2aaeac9c5641c390e498b6a4b69270 in term 0.
I20260924 19:36:28.090623 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.087888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38919 (local address 127.0.98.2:40963)
0924 19:36:28.087957 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.087959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.088154 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:36:28.088210 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.088211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.088223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.088273 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:28.088411 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.088864 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.089787 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.090161 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.090224 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.090234 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.090525 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.090563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:36:28.090617 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.087795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40963 (local address 127.0.98.3:38919)
0924 19:36:28.088046 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:28.088055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:28.088148 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:28.088279 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:28.088280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:28.088281 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:28.088351 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.088353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.088942 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.088944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.089701 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:28.089703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.090369 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.090370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:28.090451 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.090455 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:28.090503 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:36:28.090539 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260924 19:36:28.090929 1308 leader_election.cc:302] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d2aaeac9c5641c390e498b6a4b69270, bf6a21d3beac48e7b6fdb181b03a5d89; no voters:
I20260924 19:36:28.090950 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7f0fa06c5f45b2aabebbed15a2a7dd" candidate_uuid: "7d2aaeac9c5641c390e498b6a4b69270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c615b5d7a481aaa8b7b486afed2ee" is_pre_election: true
I20260924 19:36:28.091080 1241 raft_consensus.cc:2454] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2aaeac9c5641c390e498b6a4b69270 in term 0.
I20260924 19:36:28.091092 1444 raft_consensus.cc:2781] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:28.091161 1444 raft_consensus.cc:482] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:28.091187 1444 raft_consensus.cc:3037] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:28.092048 1444 raft_consensus.cc:504] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
I20260924 19:36:28.092173 1444 leader_election.cc:288] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [CANDIDATE]: Term 1 election: Requested vote from peers 206c615b5d7a481aaa8b7b486afed2ee (127.0.98.2:40963), bf6a21d3beac48e7b6fdb181b03a5d89 (127.0.98.1:38837)
I20260924 19:36:28.092332 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7f0fa06c5f45b2aabebbed15a2a7dd" candidate_uuid: "7d2aaeac9c5641c390e498b6a4b69270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c615b5d7a481aaa8b7b486afed2ee"
I20260924 19:36:28.092340 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7f0fa06c5f45b2aabebbed15a2a7dd" candidate_uuid: "7d2aaeac9c5641c390e498b6a4b69270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89"
I20260924 19:36:28.092407 1107 raft_consensus.cc:3037] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:28.092402 1241 raft_consensus.cc:3037] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:28.093072 1107 raft_consensus.cc:2454] T ad7f0fa06c5f45b2aabebbed15a2a7dd P bf6a21d3beac48e7b6fdb181b03a5d89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d2aaeac9c5641c390e498b6a4b69270 in term 1.
I20260924 19:36:28.093080 1241 raft_consensus.cc:2454] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d2aaeac9c5641c390e498b6a4b69270 in term 1.
I20260924 19:36:28.093214 1308 leader_election.cc:302] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d2aaeac9c5641c390e498b6a4b69270, bf6a21d3beac48e7b6fdb181b03a5d89; no voters:
I20260924 19:36:28.093305 1444 raft_consensus.cc:2781] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:28.093453 1444 raft_consensus.cc:686] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [term 1 LEADER]: Becoming Leader. State: Replica: 7d2aaeac9c5641c390e498b6a4b69270, State: Running, Role: LEADER
I20260924 19:36:28.093541 1444 consensus_queue.cc:237] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } }
W20260924 19:36:28.093713 1288 tablet.cc:2369] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 206c615b5d7a481aaa8b7b486afed2ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:36:28.094347 962 catalog_manager.cc:5495] T ad7f0fa06c5f45b2aabebbed15a2a7dd P 7d2aaeac9c5641c390e498b6a4b69270 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d2aaeac9c5641c390e498b6a4b69270 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7d2aaeac9c5641c390e498b6a4b69270" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206c615b5d7a481aaa8b7b486afed2ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf6a21d3beac48e7b6fdb181b03a5d89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d2aaeac9c5641c390e498b6a4b69270" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } health_report { overall_health: HEALTHY } } }
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:28.108444 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.102199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.98.1:38837)
0924 19:36:28.102264 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.102266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.102306 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:36:28.102367 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.102368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.102377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.102421 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:28.102876 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.103340 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.103998 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.104178 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.104227 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.106215 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:28.106225 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:28.106225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:28.106231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:28.107525 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.107804 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.107807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.107807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.107868 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.108039 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.108039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.108039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.108166 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:28.108258 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.108310 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.108363 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:28.130113 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.123845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.0.98.1:38837)
0924 19:36:28.123906 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.123908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.124056 (+ 148us) server_negotiation.cc:407] Connection header received
0924 19:36:28.124108 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.124109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.124118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.124166 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:28.124683 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.125154 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.125820 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.126006 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.126052 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.128096 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:28.128098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:28.128098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:28.128105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:28.129236 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.129524 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.129524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.129525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.129579 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.129708 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.129709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.129709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.129793 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:28.129854 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.129984 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.130038 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:28.151412 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:28.144737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.0.98.1:38837)
0924 19:36:28.144803 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:28.144805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:28.144881 (+ 76us) server_negotiation.cc:407] Connection header received
0924 19:36:28.144962 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:28.144964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:28.144975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:28.145023 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:28.145659 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.146159 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.146944 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:28.147124 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:28.147186 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:28.149348 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:28.149350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:28.149350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:28.149358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:28.150437 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.150773 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.150774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.150775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.150822 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:28.150973 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:28.150973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:28.150974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:28.151069 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:28.151165 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:28.151249 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:36:28.151317 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 19:36:28.151816 1067 tablet_service.cc:2908] Rejecting scan request for tablet ad7f0fa06c5f45b2aabebbed15a2a7dd: Uninitialized: clean time has not yet been initialized
I20260924 19:36:28.152603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1024
I20260924 19:36:28.156450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1158
I20260924 19:36:28.160720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1292
I20260924 19:36:28.164075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 930
2026-09-24T19:36:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1795 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[1462](info): set up 2 sockets
Sep 24 19:36:28 dist-test-slave-6kf3 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260924 19:36:29.177915 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 19:36:29.378916 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.201s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 19:36:29.521284 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.142s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 19:36:29.729408 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.208s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/test-user.keytab.
W20260924 19:36:30.536197 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.807s user 0.000s sys 0.001s
Sep 24 19:36:30 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 19:36:31.380963 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.825s user 0.000s sys 0.002s
I20260924 19:36:31.381278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:31.459251 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:31.459468 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:31.459489 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:31.460613 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:31.460646 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:36:31.460659 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:31.460671 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:31.460686 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:31.460697 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:31.462146 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34663
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:31.462354 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:31.462538 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:31.467298 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:31.467238 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:31.467371 1475 server_base.cc:1034] running on GCE node
W20260924 19:36:31.467319 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:31.467721 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:31.468021 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:31.469178 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790278591469156 us; error 34 us; skew 500 ppm
Sep 24 19:36:31 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:31.477070 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:31.477521 1475 webserver.cc:460] Webserver started at http://127.0.98.62:33957/ using document root <none> and password file <none>
I20260924 19:36:31.477725 1475 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:31.477764 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:31.477860 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:31.478863 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "079913ac08d542438479a678ac05a547"
format_stamp: "Formatted at 2026-09-24 19:36:31 on dist-test-slave-6kf3"
server_key: "44def3d94f2c34f7663fddc9d4d5178f"
server_key_iv: "0a8c75e30bfcd66ed0640e5a4a57023e"
server_key_version: "encryptionkey@0"
I20260924 19:36:31.479152 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "079913ac08d542438479a678ac05a547"
format_stamp: "Formatted at 2026-09-24 19:36:31 on dist-test-slave-6kf3"
server_key: "44def3d94f2c34f7663fddc9d4d5178f"
server_key_iv: "0a8c75e30bfcd66ed0640e5a4a57023e"
server_key_version: "encryptionkey@0"
I20260924 19:36:31.480378 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:31.481033 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:31.481177 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:31.481245 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "079913ac08d542438479a678ac05a547"
format_stamp: "Formatted at 2026-09-24 19:36:31 on dist-test-slave-6kf3"
server_key: "44def3d94f2c34f7663fddc9d4d5178f"
server_key_iv: "0a8c75e30bfcd66ed0640e5a4a57023e"
server_key_version: "encryptionkey@0"
I20260924 19:36:31.481310 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:31.490653 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:31.492748 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:31.492871 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:31.497210 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34663
I20260924 19:36:31.497260 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34663 every 8 connection(s)
I20260924 19:36:31.497575 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:31.498194 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:31.500556 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Bootstrap starting.
I20260924 19:36:31.501166 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:31.501453 1553 log.cc:824] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:31.502220 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: No bootstrap required, opened a new log
I20260924 19:36:31.504281 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } }
I20260924 19:36:31.504421 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:31.504442 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079913ac08d542438479a678ac05a547, State: Initialized, Role: FOLLOWER
I20260924 19:36:31.504557 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } }
I20260924 19:36:31.504627 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:31.504654 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:31.504685 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:31.506096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1475
I20260924 19:36:31.506168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:31.506276 392 external_mini_cluster.cc:1428] Setting key 6ef4d9f365061edd4c15f7e3feff3da5
Sep 24 19:36:31 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:31.514249 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:31.506690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.0.98.62:34663)
0924 19:36:31.506860 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:31.506868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:31.506896 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:36:31.506992 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:31.506998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:31.507040 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:31.507149 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:31.507644 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:31.508576 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:31.509272 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:31.509664 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:31.509735 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:31.511737 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:31.511741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:31.511741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:31.511751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:31.513117 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:31.513394 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:31.513396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:31.513397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:31.513472 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:31.513590 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:31.513590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:31.513591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:31.513799 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:31.513911 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:31.513978 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:36:31.514074 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260924 19:36:31.555279 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } }
I20260924 19:36:31.555457 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 079913ac08d542438479a678ac05a547; no voters:
I20260924 19:36:31.555663 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:31.555734 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:31.555948 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [term 1 LEADER]: Becoming Leader. State: Replica: 079913ac08d542438479a678ac05a547, State: Running, Role: LEADER
I20260924 19:36:31.555985 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:31.556125 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } }
I20260924 19:36:31.556555 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "079913ac08d542438479a678ac05a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } } }
I20260924 19:36:31.556663 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:31.557171 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:31.556890 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 079913ac08d542438479a678ac05a547. Latest consensus state: current_term: 1 leader_uuid: "079913ac08d542438479a678ac05a547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079913ac08d542438479a678ac05a547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34663 } } }
I20260924 19:36:31.557575 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:31.557686 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:31.559294 1567 catalog_manager.cc:1347] Generated new cluster ID: eb414bf56ef34c19b963ada528cdf51c
I20260924 19:36:31.559352 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:31.566028 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:31.566557 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:31.577484 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Generated new TSK 0
I20260924 19:36:31.577670 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 19:36:32.365866 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.785s user 0.001s sys 0.001s
I20260924 19:36:32.366169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34663
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:32.446560 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:32.446766 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:32.446790 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:32.448047 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:32.448088 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:32.448109 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:32.448129 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:32.449677 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:32.449892 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:32.450120 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:32.454975 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:32.455020 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:32.455001 1579 server_base.cc:1034] running on GCE node
W20260924 19:36:32.454975 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:32.455345 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:32.455564 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:32.456712 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790278592456691 us; error 36 us; skew 500 ppm
Sep 24 19:36:32 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:32.459600 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:32.460160 1579 webserver.cc:460] Webserver started at http://127.0.98.1:38915/ using document root <none> and password file <none>
I20260924 19:36:32.460425 1579 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:32.460479 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:32.460604 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:32.461889 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "1236539bcac74c49978ad2bbcb535615"
format_stamp: "Formatted at 2026-09-24 19:36:32 on dist-test-slave-6kf3"
server_key: "bfbd35c66a81b7cc334ea43a85c624dd"
server_key_iv: "9892ae58d0fa3e96e7337159fab959e0"
server_key_version: "encryptionkey@0"
I20260924 19:36:32.462194 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1236539bcac74c49978ad2bbcb535615"
format_stamp: "Formatted at 2026-09-24 19:36:32 on dist-test-slave-6kf3"
server_key: "bfbd35c66a81b7cc334ea43a85c624dd"
server_key_iv: "9892ae58d0fa3e96e7337159fab959e0"
server_key_version: "encryptionkey@0"
I20260924 19:36:32.463428 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:32.464120 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:32.464262 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:32.464344 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "1236539bcac74c49978ad2bbcb535615"
format_stamp: "Formatted at 2026-09-24 19:36:32 on dist-test-slave-6kf3"
server_key: "bfbd35c66a81b7cc334ea43a85c624dd"
server_key_iv: "9892ae58d0fa3e96e7337159fab959e0"
server_key_version: "encryptionkey@0"
I20260924 19:36:32.464397 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:32.474864 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:32.476763 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:32.476868 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:32.477311 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:32.477349 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:32.477377 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:32.477402 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:32.483652 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34123
I20260924 19:36:32.483748 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34123 every 8 connection(s)
Sep 24 19:36:32 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:32.484102 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:32.486794 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:32.477745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39177 (local address 127.0.98.62:34663)
0924 19:36:32.477882 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:32.477885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:32.477987 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:36:32.478509 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:32.478510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:32.478525 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:32.478609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:32.479393 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.480240 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.481675 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.481989 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.482065 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:32.484569 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:32.484571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:32.484571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:32.484579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:32.485778 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:32.486125 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:32.486125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:32.486126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:32.486174 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:32.486342 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:32.486342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:32.486342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:32.486437 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:32.486481 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:32.486673 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:36:32.486727 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 19:36:32.486974 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:32.477726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.1:39177)
0924 19:36:32.477866 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:32.477904 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:32.478404 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:32.478707 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:32.478711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:32.479027 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:32.479301 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:32.479307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.480328 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.480328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:32.481585 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:32.481589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.482068 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.482069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:32.482170 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:32.482690 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:32.482702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:32.484481 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:32.485839 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:32.485844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:32.485846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:32.486053 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:32.486217 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:32.486218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:32.486218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:32.486303 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:32.486477 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:32.486479 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:32.486585 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:36:32.486661 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:36:32.486771 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260924 19:36:32.487409 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34663
I20260924 19:36:32.487489 1711 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:32.487697 1711 heartbeater.cc:511] Master 127.0.98.62:34663 requested a full tablet report, sending...
I20260924 19:36:32.488233 1506 ts_manager.cc:194] Registered new tserver with Master: 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123)
I20260924 19:36:32.488884 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39177
I20260924 19:36:32.490924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1579
I20260924 19:36:32.491011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:32.491175 392 external_mini_cluster.cc:1428] Setting key 95971fec40ab9de619648e10afec0ef7
I20260924 19:36:32.493757 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:32.488382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.1:43995)
0924 19:36:32.488493 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:32.488502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:32.488559 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:32.488664 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:32.488664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:32.488820 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:32.488867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:32.488868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.489658 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.489658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:32.490160 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:32.490162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.490505 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.490506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:32.490549 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:32.491001 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:32.491006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:32.491327 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:32.492839 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:32.492840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:32.492840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:32.493057 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:32.493275 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:32.493275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:32.493275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:32.493325 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:32.493542 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:32.493542 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:32.493575 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:36:32.493617 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:36:32.493686 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
I20260924 19:36:32.493796 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:32.488406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43995 (local address 127.0.98.62:34663)
0924 19:36:32.488442 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:32.488443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:32.488557 (+ 114us) server_negotiation.cc:407] Connection header received
0924 19:36:32.488608 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:32.488609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:32.488619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:32.488662 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:32.488927 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.489569 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.490252 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:32.490440 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:32.490506 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:32.491408 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:32.491409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:32.491410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:32.491416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:32.492763 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:32.493137 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:32.493138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:32.493139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:32.493202 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:32.493372 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:32.493373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:32.493373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:32.493487 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:32.493552 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:32.493647 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:36:32.493719 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 19:36:33.097332 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.605s user 0.002s sys 0.000s
I20260924 19:36:33.097679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34663
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:33.177762 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:33.177963 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:33.177986 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:33.179076 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:33.179111 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:33.179131 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:33.179145 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:33.180632 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:33.180817 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:33.180997 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:33.185720 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:33.185897 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:33.185989 1720 server_base.cc:1034] running on GCE node
W20260924 19:36:33.185776 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:33.186241 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:33.186456 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:33.187597 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790278593187581 us; error 27 us; skew 500 ppm
Sep 24 19:36:33 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:33.190032 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:33.190434 1720 webserver.cc:460] Webserver started at http://127.0.98.2:34879/ using document root <none> and password file <none>
I20260924 19:36:33.190606 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:33.190644 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:33.190730 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:33.191738 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf278f11f4dc463b985864225944d329"
format_stamp: "Formatted at 2026-09-24 19:36:33 on dist-test-slave-6kf3"
server_key: "9d5da2d634af79ad0202bfa13a4d543a"
server_key_iv: "cb7fbbd37ecd69b96515cd03b6e839cd"
server_key_version: "encryptionkey@0"
I20260924 19:36:33.192103 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf278f11f4dc463b985864225944d329"
format_stamp: "Formatted at 2026-09-24 19:36:33 on dist-test-slave-6kf3"
server_key: "9d5da2d634af79ad0202bfa13a4d543a"
server_key_iv: "cb7fbbd37ecd69b96515cd03b6e839cd"
server_key_version: "encryptionkey@0"
I20260924 19:36:33.193336 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:33.194008 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:33.194142 1720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:33.194221 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "bf278f11f4dc463b985864225944d329"
format_stamp: "Formatted at 2026-09-24 19:36:33 on dist-test-slave-6kf3"
server_key: "9d5da2d634af79ad0202bfa13a4d543a"
server_key_iv: "cb7fbbd37ecd69b96515cd03b6e839cd"
server_key_version: "encryptionkey@0"
I20260924 19:36:33.194276 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:33.219421 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:33.221501 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:33.221633 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:33.222155 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:33.222204 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:33.222236 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:33.222259 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:36:33 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:33.228922 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46341
I20260924 19:36:33.229002 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46341 every 8 connection(s)
I20260924 19:36:33.229332 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:33.231698 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:33.222626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56723 (local address 127.0.98.62:34663)
0924 19:36:33.222727 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:33.222729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:33.223286 (+ 557us) server_negotiation.cc:407] Connection header received
0924 19:36:33.223556 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:33.223558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:33.223572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:33.223650 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:33.224580 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.225149 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.226604 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.226843 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.226916 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:33.229274 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:33.229276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:33.229277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:33.229284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:33.230629 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:33.230966 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:33.230967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:33.230968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:33.231014 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:33.231181 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:33.231181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:33.231182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:33.231278 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:33.231322 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:33.231571 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:36:33.231630 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260924 19:36:33.231842 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:33.222629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.2:56723)
0924 19:36:33.222773 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:33.222816 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:33.223432 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:33.223792 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:33.223797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:33.224188 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:33.224484 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:33.224491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.225226 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.225227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:33.226529 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:33.226532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.226937 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.226939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:33.227075 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:33.227543 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:33.227552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:33.229146 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:33.230691 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:33.230695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:33.230697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:33.230910 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:33.231056 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:33.231057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:33.231057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:33.231142 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:33.231332 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:33.231335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:33.231470 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:36:33.231551 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:36:33.231668 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260924 19:36:33.232302 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34663
I20260924 19:36:33.232385 1852 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:33.232414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1720
I20260924 19:36:33.232508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:33.232600 1852 heartbeater.cc:511] Master 127.0.98.62:34663 requested a full tablet report, sending...
I20260924 19:36:33.232681 392 external_mini_cluster.cc:1428] Setting key b77788fc1e8553872828958b10677e10
I20260924 19:36:33.232985 1506 ts_manager.cc:194] Registered new tserver with Master: bf278f11f4dc463b985864225944d329 (127.0.98.2:46341)
I20260924 19:36:33.233423 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56723
I20260924 19:36:33.239832 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:33.233329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44729 (local address 127.0.98.62:34663)
0924 19:36:33.233364 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:33.233366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:33.234668 (+ 1302us) server_negotiation.cc:407] Connection header received
0924 19:36:33.234708 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:33.234709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:33.234717 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:33.234760 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:33.235100 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.235589 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.236528 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.236715 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.236766 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:33.237662 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:33.237663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:33.237663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:33.237669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:33.238824 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:33.239169 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:33.239170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:33.239170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:33.239212 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:33.239399 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:33.239400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:33.239400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:33.239521 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:33.239569 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:33.239717 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:36:33.239771 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:36:33.239984 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:33.234447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.2:44729)
0924 19:36:33.234585 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:33.234597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:33.234661 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:33.234765 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:33.234766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:33.234964 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:33.235032 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:33.235034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.235677 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.235678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:33.236459 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:33.236461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:33.236781 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:33.236782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:33.236843 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:33.237229 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:33.237234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:33.237590 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:33.238897 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:33.238898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:33.238899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:33.239104 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:33.239259 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:33.239260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:33.239261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:33.239318 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:33.239580 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:33.239581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:33.239626 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:36:33.239701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:36:33.239793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 19:36:33.489837 1711 heartbeater.cc:503] Master 127.0.98.62:34663 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 19:36:34.145623 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.912s user 0.001s sys 0.001s
I20260924 19:36:34.145936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34663
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:34.223481 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:34.223671 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:34.223693 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:34.224797 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:34.224835 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:34.224853 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:34.224869 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:34.226286 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:34.226462 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:34.226634 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:34.231707 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:34.231684 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:34.231817 1861 server_base.cc:1034] running on GCE node
W20260924 19:36:34.231678 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:34.232156 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:34.232375 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:34.233495 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790278594233486 us; error 25 us; skew 500 ppm
I20260924 19:36:34.234200 1852 heartbeater.cc:503] Master 127.0.98.62:34663 was elected leader, sending a full tablet report...
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:34.235912 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:34.236308 1861 webserver.cc:460] Webserver started at http://127.0.98.3:36933/ using document root <none> and password file <none>
I20260924 19:36:34.236476 1861 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:34.236513 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:34.236598 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:34.237560 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "a55e7b49240546f9bf83763b0f43230f"
format_stamp: "Formatted at 2026-09-24 19:36:34 on dist-test-slave-6kf3"
server_key: "6bf4937221a1a72c2eb4e997f5825c08"
server_key_iv: "c4fc6a5449b71cb5daf553c5fe22362a"
server_key_version: "encryptionkey@0"
I20260924 19:36:34.237850 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a55e7b49240546f9bf83763b0f43230f"
format_stamp: "Formatted at 2026-09-24 19:36:34 on dist-test-slave-6kf3"
server_key: "6bf4937221a1a72c2eb4e997f5825c08"
server_key_iv: "c4fc6a5449b71cb5daf553c5fe22362a"
server_key_version: "encryptionkey@0"
I20260924 19:36:34.238960 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:36:34.239544 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.239697 1861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.239775 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "a55e7b49240546f9bf83763b0f43230f"
format_stamp: "Formatted at 2026-09-24 19:36:34 on dist-test-slave-6kf3"
server_key: "6bf4937221a1a72c2eb4e997f5825c08"
server_key_iv: "c4fc6a5449b71cb5daf553c5fe22362a"
server_key_version: "encryptionkey@0"
I20260924 19:36:34.239833 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:34.255646 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:34.257699 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:34.257877 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:34.258363 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:34.258402 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.258430 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:34.258447 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.264515 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36683
I20260924 19:36:34.264587 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36683 every 8 connection(s)
I20260924 19:36:34.264874 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:34.268167 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.258768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49313 (local address 127.0.98.62:34663)
0924 19:36:34.258895 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.258897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.259089 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:36:34.259551 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.259552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.259568 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.259652 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.260475 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.261220 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.262697 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.262995 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.263075 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.265444 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.265445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.265446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.265453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.267058 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.267424 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.267425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.267425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.267468 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.267649 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.267649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.267650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.267751 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.267801 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.268041 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.268100 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260924 19:36:34.268342 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.258761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.3:49313)
0924 19:36:34.258968 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.259007 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.259461 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.259715 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.259720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.260093 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:34.260376 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.260384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.261314 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.261315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.262604 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.262608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.263079 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.263080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.263186 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.263699 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:34.263710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:34.265365 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:34.267123 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.267128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.267130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.267370 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.267509 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.267509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.267509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.267611 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.267801 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:34.267804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:34.267916 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.268027 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.268143 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
I20260924 19:36:34.268761 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34663
I20260924 19:36:34.268838 1993 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:34.269007 1993 heartbeater.cc:511] Master 127.0.98.62:34663 requested a full tablet report, sending...
I20260924 19:36:34.269375 1506 ts_manager.cc:194] Registered new tserver with Master: a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683)
I20260924 19:36:34.269709 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49313
I20260924 19:36:34.270171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 1861
I20260924 19:36:34.270251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:34.270417 392 external_mini_cluster.cc:1428] Setting key 41deb9580b8b8d06049ec3bddfa87622
I20260924 19:36:34.271119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:36:34.274859 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.269763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52641 (local address 127.0.98.62:34663)
0924 19:36:34.269801 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.269802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.269858 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:36:34.269914 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.269915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.269924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.269985 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.270313 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.271004 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.271652 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.271852 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.271904 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.272755 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.272756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.272757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.272763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.273954 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.274230 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.274231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.274231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.274280 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.274432 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.274433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.274433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.274530 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.274580 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.274703 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.274775 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:34.274868 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.269704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.98.3:52641)
0924 19:36:34.269809 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.269819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.269869 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.269989 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.269989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.270166 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:34.270224 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.270226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.271107 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.271108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.271587 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.271588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.271906 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.271907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.271984 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.272344 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:34.272348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:34.272684 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:34.274025 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.274026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.274026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.274174 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.274330 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.274331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.274331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.274385 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.274579 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:34.274579 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:34.274617 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.274677 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.274768 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":63,"threads_started":1}
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:34.291199 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.285482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.98.62:34663)
0924 19:36:34.285534 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.285536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.285547 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:36:34.285605 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.285605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.285613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.285663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.286140 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.286598 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.287318 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.287501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.287550 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.289459 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.289460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.289460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.289467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.290437 (+ 970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.290702 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.290703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.290703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.290754 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.290872 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.290873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.290873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.290957 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.291020 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.291086 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.291136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:36:34.292740 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:34.294230 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:34.300304 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.0.1:33592)
0924 19:36:34.296789 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.296797 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.296844 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.297167 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.297167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.297169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.297197 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.297198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298333 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.298334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.299185 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.299188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.300040 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.300102 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.300113 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.300161 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.300204 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260924 19:36:34.300309 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.0.98.3:36683)
0924 19:36:34.296973 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.296977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.296985 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:34.297016 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.297018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.297101 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.297170 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.297248 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298279 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.299255 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.299968 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300031 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.300105 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.300175 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.300211 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 19:36:34.300469 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34123 (local address 127.0.0.1:34010)
0924 19:36:34.296510 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.296553 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.296614 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.296939 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.296942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.296944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.297006 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.297008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298297 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.298298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.299368 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.299369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.300217 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.300272 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.300275 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.300322 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.300358 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":27,"thread_start_us":90,"threads_started":1}
I20260924 19:36:34.300639 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.0.98.1:34123)
0924 19:36:34.296719 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.296725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.296735 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:34.296770 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.296772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.296789 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.296862 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.297081 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298215 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.299446 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.300150 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300212 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.300273 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.300524 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.300567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260924 19:36:34.301285 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.0.98.2:46341)
0924 19:36:34.296851 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.296856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.296865 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:34.296898 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.296900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.296916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.296983 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.297326 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298030 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300306 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.300891 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.300959 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.301014 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.301162 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.301203 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":62,"threads_started":1}
I20260924 19:36:34.301316 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b (DEFAULT_TABLE table=test-table [id=c8364cda31b2466a99cb35be7959a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.301541 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b (DEFAULT_TABLE table=test-table [id=c8364cda31b2466a99cb35be7959a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.301692 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce695208a6f4627a5ae30f0a6d4b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.301826 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce695208a6f4627a5ae30f0a6d4b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.304869 2017 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Bootstrap starting.
I20260924 19:36:34.305012 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.296377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46341 (local address 127.0.0.1:49590)
0924 19:36:34.296711 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.296719 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.296758 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.297046 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.297047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.297049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.297091 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.297092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.298111 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.298111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.299356 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.299359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.301030 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.301031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.301092 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.301096 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.304895 (+ 3799us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.304942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":85,"threads_started":1}
I20260924 19:36:34.305265 2018 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Bootstrap starting.
I20260924 19:36:34.305562 2017 tablet_bootstrap.cc:654] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.305847 2018 tablet_bootstrap.cc:654] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.305878 2017 log.cc:824] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:34.305891 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b (DEFAULT_TABLE table=test-table [id=c8364cda31b2466a99cb35be7959a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.306133 2018 log.cc:824] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:34.306198 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce695208a6f4627a5ae30f0a6d4b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.306574 2017 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: No bootstrap required, opened a new log
I20260924 19:36:34.306630 2017 ts_tablet_manager.cc:1397] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:34.306715 2018 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: No bootstrap required, opened a new log
I20260924 19:36:34.306766 2018 ts_tablet_manager.cc:1397] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:34.308240 2018 raft_consensus.cc:348] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.308367 2018 raft_consensus.cc:374] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.308396 2018 raft_consensus.cc:729] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55e7b49240546f9bf83763b0f43230f, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.308526 2018 consensus_queue.cc:260] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.308772 2018 ts_tablet_manager.cc:1428] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:36:34.308895 1993 heartbeater.cc:503] Master 127.0.98.62:34663 was elected leader, sending a full tablet report...
I20260924 19:36:34.309263 2022 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Bootstrap starting.
I20260924 19:36:34.309185 2017 raft_consensus.cc:348] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.309329 2017 raft_consensus.cc:374] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.309371 2017 raft_consensus.cc:729] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1236539bcac74c49978ad2bbcb535615, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.309474 2017 consensus_queue.cc:260] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.309680 2017 ts_tablet_manager.cc:1428] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:34.309903 2022 tablet_bootstrap.cc:654] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.310165 2022 log.cc:824] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:34.310711 2022 tablet_bootstrap.cc:492] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: No bootstrap required, opened a new log
I20260924 19:36:34.310767 2022 ts_tablet_manager.cc:1397] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:34.312685 2022 raft_consensus.cc:348] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.312806 2022 raft_consensus.cc:374] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.312826 2022 raft_consensus.cc:729] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf278f11f4dc463b985864225944d329, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.312929 2022 consensus_queue.cc:260] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.313127 2022 ts_tablet_manager.cc:1428] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:36:34.336776 2023 raft_consensus.cc:482] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:34.336898 2023 raft_consensus.cc:504] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.337248 2023 leader_election.cc:288] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683), bf278f11f4dc463b985864225944d329 (127.0.98.2:46341)
I20260924 19:36:34.341224 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.337478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46663 (local address 127.0.98.3:36683)
0924 19:36:34.337521 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.337523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.337672 (+ 149us) server_negotiation.cc:407] Connection header received
0924 19:36:34.337713 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.337714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.337727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.337789 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.337997 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.338735 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.340316 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.340886 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.340949 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.340959 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.341118 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.341160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:34.341276 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.337515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55195 (local address 127.0.98.2:46341)
0924 19:36:34.337557 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.337559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.337665 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:36:34.337714 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.337715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.337726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.337790 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.337996 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.338730 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.340310 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.340877 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.340949 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.340959 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.341160 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.341199 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:36:34.341257 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.337406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.1:46663)
0924 19:36:34.337589 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.337600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.337665 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.337824 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.337825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.337828 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.337930 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.337933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.338822 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.338823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.340232 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.340236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.340975 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.340976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.341047 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.341053 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.341101 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.341143 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":77,"thread_start_us":112,"threads_started":1}
I20260924 19:36:34.341498 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.337432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46341 (local address 127.0.98.1:55195)
0924 19:36:34.337600 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.337609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.337666 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.337824 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.337824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.337827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.337930 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.337933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.338827 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.338828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.340233 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.340236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.341011 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.341012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.341088 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.341092 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.341145 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.341188 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":95,"thread_start_us":102,"threads_started":1}
I20260924 19:36:34.341612 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce695208a6f4627a5ae30f0a6d4b69b" candidate_uuid: "1236539bcac74c49978ad2bbcb535615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55e7b49240546f9bf83763b0f43230f" is_pre_election: true
I20260924 19:36:34.341782 1947 raft_consensus.cc:2454] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1236539bcac74c49978ad2bbcb535615 in term 0.
I20260924 19:36:34.341851 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce695208a6f4627a5ae30f0a6d4b69b" candidate_uuid: "1236539bcac74c49978ad2bbcb535615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf278f11f4dc463b985864225944d329" is_pre_election: true
I20260924 19:36:34.342005 1806 raft_consensus.cc:2454] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1236539bcac74c49978ad2bbcb535615 in term 0.
I20260924 19:36:34.342038 1597 leader_election.cc:302] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1236539bcac74c49978ad2bbcb535615, a55e7b49240546f9bf83763b0f43230f; no voters:
I20260924 19:36:34.342168 2023 raft_consensus.cc:2781] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:34.342208 2023 raft_consensus.cc:482] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:34.342228 2023 raft_consensus.cc:3037] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.342902 2023 raft_consensus.cc:504] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.343022 2023 leader_election.cc:288] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [CANDIDATE]: Term 1 election: Requested vote from peers a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683), bf278f11f4dc463b985864225944d329 (127.0.98.2:46341)
I20260924 19:36:34.343173 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce695208a6f4627a5ae30f0a6d4b69b" candidate_uuid: "1236539bcac74c49978ad2bbcb535615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55e7b49240546f9bf83763b0f43230f"
I20260924 19:36:34.343214 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce695208a6f4627a5ae30f0a6d4b69b" candidate_uuid: "1236539bcac74c49978ad2bbcb535615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf278f11f4dc463b985864225944d329"
I20260924 19:36:34.343245 1947 raft_consensus.cc:3037] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.343278 1806 raft_consensus.cc:3037] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.343971 1947 raft_consensus.cc:2454] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1236539bcac74c49978ad2bbcb535615 in term 1.
I20260924 19:36:34.343987 1806 raft_consensus.cc:2454] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1236539bcac74c49978ad2bbcb535615 in term 1.
I20260924 19:36:34.344120 1596 leader_election.cc:302] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1236539bcac74c49978ad2bbcb535615, bf278f11f4dc463b985864225944d329; no voters:
I20260924 19:36:34.344239 2023 raft_consensus.cc:2781] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:34.344385 2023 raft_consensus.cc:686] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 1 LEADER]: Becoming Leader. State: Replica: 1236539bcac74c49978ad2bbcb535615, State: Running, Role: LEADER
I20260924 19:36:34.344470 2023 consensus_queue.cc:237] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } }
I20260924 19:36:34.345249 1506 catalog_manager.cc:5495] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1236539bcac74c49978ad2bbcb535615 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1236539bcac74c49978ad2bbcb535615" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:34.357898 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.355535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.0.98.1:34123)
0924 19:36:34.355587 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.355589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.355618 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:36:34.355710 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.355711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.355727 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.355779 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:34.355953 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.356460 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.357249 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.357436 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.357488 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.357638 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.357729 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.357776 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.357795 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.357810 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:36:34.359591 1947 raft_consensus.cc:1261] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Refusing update from remote peer 1236539bcac74c49978ad2bbcb535615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.359598 1806 raft_consensus.cc:1261] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Refusing update from remote peer 1236539bcac74c49978ad2bbcb535615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.359908 2028 consensus_queue.cc:1035] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.360066 2023 consensus_queue.cc:1035] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.362224 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332981122494791680 to 7332981122435776512. Current Snapshot: MvccSnapshot[applied={T|T < 7332981122494791680}]
I20260924 19:36:34.362699 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332981122494791680 to 7332981122435776512. Current Snapshot: MvccSnapshot[applied={T|T < 7332981122494791680}]
I20260924 19:36:34.365939 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:34.366019 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:34.366683 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Sending DeleteTablet for 3 replicas of tablet 5ce695208a6f4627a5ae30f0a6d4b69b
I20260924 19:36:34.366977 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49590
I20260924 19:36:34.367187 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33592
I20260924 19:36:34.367226 2041 tablet_replica.cc:330] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: stopping tablet replica
I20260924 19:36:34.367194 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:36:34.367321 2041 raft_consensus.cc:2229] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 19:36:34.367322 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:34.367410 2041 raft_consensus.cc:2258] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.367619 2042 tablet_replica.cc:330] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: stopping tablet replica
I20260924 19:36:34.367715 2042 raft_consensus.cc:2229] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:34.367805 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce695208a6f4627a5ae30f0a6d4b69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34010
I20260924 19:36:34.368336 2041 ts_tablet_manager.cc:1905] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.368657 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 17c629b7a476426f94db838ce7c06203 (DEFAULT_TABLE table=test-table [id=896d083b6f0c4ccba8351ae50c0d1342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.368690 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 17c629b7a476426f94db838ce7c06203 (DEFAULT_TABLE table=test-table [id=896d083b6f0c4ccba8351ae50c0d1342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.368780 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c629b7a476426f94db838ce7c06203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.368779 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c629b7a476426f94db838ce7c06203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.370069 2022 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Bootstrap starting.
I20260924 19:36:34.370119 2018 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Bootstrap starting.
I20260924 19:36:34.370473 2022 tablet_bootstrap.cc:654] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.370502 2018 tablet_bootstrap.cc:654] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.370535 2041 ts_tablet_manager.cc:1918] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:34.370594 2041 log.cc:1196] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/5ce695208a6f4627a5ae30f0a6d4b69b
I20260924 19:36:34.370841 2041 ts_tablet_manager.cc:1939] T 5ce695208a6f4627a5ae30f0a6d4b69b P bf278f11f4dc463b985864225944d329: Deleting consensus metadata
I20260924 19:36:34.371011 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 17c629b7a476426f94db838ce7c06203 (DEFAULT_TABLE table=test-table [id=896d083b6f0c4ccba8351ae50c0d1342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:34.371121 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c629b7a476426f94db838ce7c06203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.371357 2042 raft_consensus.cc:2258] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.371718 2042 ts_tablet_manager.cc:1905] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.372538 2017 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Bootstrap starting.
I20260924 19:36:34.372845 2042 ts_tablet_manager.cc:1918] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:34.372901 2042 log.cc:1196] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/5ce695208a6f4627a5ae30f0a6d4b69b
I20260924 19:36:34.372907 1492 catalog_manager.cc:4841] TS bf278f11f4dc463b985864225944d329 (127.0.98.2:46341): tablet 5ce695208a6f4627a5ae30f0a6d4b69b (table test-table [id=c8364cda31b2466a99cb35be7959a600]) successfully deleted
I20260924 19:36:34.372949 2017 tablet_bootstrap.cc:654] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.373056 2043 tablet_replica.cc:330] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: stopping tablet replica
I20260924 19:36:34.373112 2042 ts_tablet_manager.cc:1939] T 5ce695208a6f4627a5ae30f0a6d4b69b P a55e7b49240546f9bf83763b0f43230f: Deleting consensus metadata
I20260924 19:36:34.373147 2043 raft_consensus.cc:2229] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:34.373262 2043 raft_consensus.cc:2258] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.373649 2022 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: No bootstrap required, opened a new log
I20260924 19:36:34.373670 1493 catalog_manager.cc:4841] TS a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683): tablet 5ce695208a6f4627a5ae30f0a6d4b69b (table test-table [id=c8364cda31b2466a99cb35be7959a600]) successfully deleted
I20260924 19:36:34.373696 2022 ts_tablet_manager.cc:1397] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:36:34.373929 2022 raft_consensus.cc:348] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.373975 2017 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: No bootstrap required, opened a new log
I20260924 19:36:34.373993 2022 raft_consensus.cc:374] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.374006 2017 ts_tablet_manager.cc:1397] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:34.374012 2022 raft_consensus.cc:729] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf278f11f4dc463b985864225944d329, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.374058 2022 consensus_queue.cc:260] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.374120 2017 raft_consensus.cc:348] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.374156 2022 ts_tablet_manager.cc:1428] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.374176 2017 raft_consensus.cc:374] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.374192 2017 raft_consensus.cc:729] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1236539bcac74c49978ad2bbcb535615, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.374228 2017 consensus_queue.cc:260] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.374326 2017 ts_tablet_manager.cc:1428] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.374541 2043 ts_tablet_manager.cc:1905] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.375901 2043 ts_tablet_manager.cc:1918] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:34.375990 2043 log.cc:1196] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/5ce695208a6f4627a5ae30f0a6d4b69b
I20260924 19:36:34.376255 2043 ts_tablet_manager.cc:1939] T 5ce695208a6f4627a5ae30f0a6d4b69b P 1236539bcac74c49978ad2bbcb535615: Deleting consensus metadata
I20260924 19:36:34.376286 2018 tablet_bootstrap.cc:492] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: No bootstrap required, opened a new log
I20260924 19:36:34.376325 2018 ts_tablet_manager.cc:1397] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 19:36:34.376451 2018 raft_consensus.cc:348] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.376497 2018 raft_consensus.cc:374] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.376507 2018 raft_consensus.cc:729] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55e7b49240546f9bf83763b0f43230f, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.376538 2018 consensus_queue.cc:260] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.376614 2018 ts_tablet_manager.cc:1428] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.376639 1493 catalog_manager.cc:4841] TS 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123): tablet 5ce695208a6f4627a5ae30f0a6d4b69b (table test-table [id=c8364cda31b2466a99cb35be7959a600]) successfully deleted
I20260924 19:36:34.454970 2021 raft_consensus.cc:482] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:34.455076 2021 raft_consensus.cc:504] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.455400 2021 leader_election.cc:288] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123), bf278f11f4dc463b985864225944d329 (127.0.98.2:46341)
I20260924 19:36:34.458554 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.455637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46691 (local address 127.0.98.2:46341)
0924 19:36:34.455682 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.455684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.455829 (+ 145us) server_negotiation.cc:407] Connection header received
0924 19:36:34.455944 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.455946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.455955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.456002 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.456135 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.456681 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.457747 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.458208 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.458276 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.458288 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.458447 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.458486 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:34.458570 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.455554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46341 (local address 127.0.98.3:46691)
0924 19:36:34.455753 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.455763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.455854 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.456008 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.456008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.456010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.456083 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.456084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.456760 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.456761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.457668 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.457670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.458294 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.458295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.458372 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.458377 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.458433 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.458472 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260924 19:36:34.458757 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.455545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34123 (local address 127.0.98.3:40439)
0924 19:36:34.455638 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.455649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.455695 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.455826 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.455827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.455829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.455894 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.455896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.456787 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.456789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.457845 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.457847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.458533 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.458534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.458607 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.458610 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.458653 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.458691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 19:36:34.458992 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.455672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40439 (local address 127.0.98.1:34123)
0924 19:36:34.455726 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.455728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.455736 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:34.455762 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.455762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.455770 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.455815 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.455987 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.456680 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.457931 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.458437 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.458495 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.458504 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.458895 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.458934 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:34.459034 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c629b7a476426f94db838ce7c06203" candidate_uuid: "a55e7b49240546f9bf83763b0f43230f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf278f11f4dc463b985864225944d329" is_pre_election: true
I20260924 19:36:34.459128 1806 raft_consensus.cc:2454] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55e7b49240546f9bf83763b0f43230f in term 0.
I20260924 19:36:34.459220 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c629b7a476426f94db838ce7c06203" candidate_uuid: "a55e7b49240546f9bf83763b0f43230f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1236539bcac74c49978ad2bbcb535615" is_pre_election: true
I20260924 19:36:34.459327 1878 leader_election.cc:302] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55e7b49240546f9bf83763b0f43230f, bf278f11f4dc463b985864225944d329; no voters:
I20260924 19:36:34.459357 1665 raft_consensus.cc:2454] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55e7b49240546f9bf83763b0f43230f in term 0.
I20260924 19:36:34.459444 2021 raft_consensus.cc:2781] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:34.459478 2021 raft_consensus.cc:482] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:34.459496 2021 raft_consensus.cc:3037] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.460184 2021 raft_consensus.cc:504] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.460307 2021 leader_election.cc:288] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [CANDIDATE]: Term 1 election: Requested vote from peers 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123), bf278f11f4dc463b985864225944d329 (127.0.98.2:46341)
I20260924 19:36:34.460456 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c629b7a476426f94db838ce7c06203" candidate_uuid: "a55e7b49240546f9bf83763b0f43230f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf278f11f4dc463b985864225944d329"
I20260924 19:36:34.460518 1806 raft_consensus.cc:3037] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.461246 1806 raft_consensus.cc:2454] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55e7b49240546f9bf83763b0f43230f in term 1.
I20260924 19:36:34.461338 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c629b7a476426f94db838ce7c06203" candidate_uuid: "a55e7b49240546f9bf83763b0f43230f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1236539bcac74c49978ad2bbcb535615"
I20260924 19:36:34.461388 1878 leader_election.cc:302] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55e7b49240546f9bf83763b0f43230f, bf278f11f4dc463b985864225944d329; no voters:
I20260924 19:36:34.461412 1665 raft_consensus.cc:3037] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.461457 2021 raft_consensus.cc:2781] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:34.461565 2021 raft_consensus.cc:686] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 1 LEADER]: Becoming Leader. State: Replica: a55e7b49240546f9bf83763b0f43230f, State: Running, Role: LEADER
I20260924 19:36:34.461637 2021 consensus_queue.cc:237] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.462234 1665 raft_consensus.cc:2454] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55e7b49240546f9bf83763b0f43230f in term 1.
I20260924 19:36:34.462328 1506 catalog_manager.cc:5495] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f reported cstate change: term changed from 0 to 1, leader changed from <none> to a55e7b49240546f9bf83763b0f43230f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a55e7b49240546f9bf83763b0f43230f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: HEALTHY } } }
W20260924 19:36:34.480386 1853 tablet.cc:2369] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:36:34.485522 1712 tablet.cc:2369] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:34.514798 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.508684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.0.98.62:34663)
0924 19:36:34.508768 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.508770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.508779 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:34.508861 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.508861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.508869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.508916 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.509426 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.509899 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.510602 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.510783 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.510823 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.512731 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.512732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.512732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.512738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.513949 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.514244 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.514245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.514245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.514288 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.514417 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.514417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.514418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.514507 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.514548 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.514646 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.514704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:36:34.514850 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.508567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34663 (local address 127.0.0.1:39000)
0924 19:36:34.508715 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.508728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.508798 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.508935 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.508935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.509311 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:34.509362 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.509365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.509965 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.509966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.510535 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.510537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.510850 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.510851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.510902 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.511357 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:34.511361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:34.512659 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:34.514006 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.514010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.514012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.514188 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.514328 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:34.514328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:34.514329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:34.514381 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:34.514545 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:34.514549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:34.514587 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.514630 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.514748 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
W20260924 19:36:34.515362 1994 tablet.cc:2369] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:36:34.516790 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39000:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:36:34.516960 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:34.518184 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 15bdd47137a241998e139135103705b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32bccba8efc646d0b374bb3786190fda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:34.518188 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 15bdd47137a241998e139135103705b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32bccba8efc646d0b374bb3786190fda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:34.518317 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15bdd47137a241998e139135103705b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.518321 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15bdd47137a241998e139135103705b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.519338 2017 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: Bootstrap starting.
I20260924 19:36:34.519351 2022 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: Bootstrap starting.
I20260924 19:36:34.519403 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 15bdd47137a241998e139135103705b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32bccba8efc646d0b374bb3786190fda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:34.519531 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15bdd47137a241998e139135103705b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:34.519670 2017 tablet_bootstrap.cc:654] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.519752 2022 tablet_bootstrap.cc:654] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.520278 2017 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: No bootstrap required, opened a new log
I20260924 19:36:34.520330 2017 ts_tablet_manager.cc:1397] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:34.520362 2022 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: No bootstrap required, opened a new log
I20260924 19:36:34.520402 2022 ts_tablet_manager.cc:1397] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:34.520449 2017 raft_consensus.cc:348] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.520514 2017 raft_consensus.cc:374] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.520532 2017 raft_consensus.cc:729] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1236539bcac74c49978ad2bbcb535615, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.520521 2022 raft_consensus.cc:348] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.520567 2022 raft_consensus.cc:374] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.520584 2022 raft_consensus.cc:729] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf278f11f4dc463b985864225944d329, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.520577 2017 consensus_queue.cc:260] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.520622 2022 consensus_queue.cc:260] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.520660 2017 ts_tablet_manager.cc:1428] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.520709 2022 ts_tablet_manager.cc:1428] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.520694 2037 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.520781 2037 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.521070 2018 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: Bootstrap starting.
I20260924 19:36:34.521242 2025 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.521329 2025 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.521440 2018 tablet_bootstrap.cc:654] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:34.521991 2018 tablet_bootstrap.cc:492] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: No bootstrap required, opened a new log
I20260924 19:36:34.522030 2018 ts_tablet_manager.cc:1397] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:34.522269 2018 raft_consensus.cc:348] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.522321 2018 raft_consensus.cc:374] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:34.522333 2018 raft_consensus.cc:729] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55e7b49240546f9bf83763b0f43230f, State: Initialized, Role: FOLLOWER
I20260924 19:36:34.522373 2018 consensus_queue.cc:260] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.522450 2018 ts_tablet_manager.cc:1428] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:34.522516 2048 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.522583 2048 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.681144 2025 raft_consensus.cc:482] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:34.681257 2025 raft_consensus.cc:504] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.681581 2025 leader_election.cc:288] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123), a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683)
I20260924 19:36:34.685671 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.681842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60599 (local address 127.0.98.1:34123)
0924 19:36:34.681900 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.681902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.682038 (+ 136us) server_negotiation.cc:407] Connection header received
0924 19:36:34.682153 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.682154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.682175 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.682237 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.682382 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.683134 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.684661 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.685276 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.685343 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.685353 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.685554 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.685601 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:36:34.685678 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.682077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42577 (local address 127.0.98.3:36683)
0924 19:36:34.682113 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.682115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.682169 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:36:34.682222 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.682223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.682232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.682274 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:34.682380 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.683127 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.684662 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.685276 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.685343 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.685353 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.685567 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.685606 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:36:34.685719 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.681999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.2:42577)
0924 19:36:34.682110 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.682119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.682168 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.682280 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.682280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.682281 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.682327 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.682328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.683195 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.683195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.684586 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.684588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.685347 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.685348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.685467 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.685469 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.685558 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.685589 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":84,"thread_start_us":62,"threads_started":1}
I20260924 19:36:34.686058 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15bdd47137a241998e139135103705b1" candidate_uuid: "bf278f11f4dc463b985864225944d329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55e7b49240546f9bf83763b0f43230f" is_pre_election: true
I20260924 19:36:34.686151 1947 raft_consensus.cc:2454] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf278f11f4dc463b985864225944d329 in term 0.
I20260924 19:36:34.686219 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.681795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34123 (local address 127.0.98.2:60599)
0924 19:36:34.681961 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.681971 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.682040 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.682243 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.682243 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.682245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:34.682320 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.682322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.683211 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.683211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.684584 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.684588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.685349 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.685350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.685449 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.685455 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.685542 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.685589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":26,"thread_start_us":103,"threads_started":1}
I20260924 19:36:34.686316 1738 leader_election.cc:302] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55e7b49240546f9bf83763b0f43230f, bf278f11f4dc463b985864225944d329; no voters:
I20260924 19:36:34.686457 2025 raft_consensus.cc:2781] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:34.686493 2025 raft_consensus.cc:482] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:34.686512 2025 raft_consensus.cc:3037] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.686561 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15bdd47137a241998e139135103705b1" candidate_uuid: "bf278f11f4dc463b985864225944d329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1236539bcac74c49978ad2bbcb535615" is_pre_election: true
I20260924 19:36:34.686651 1665 raft_consensus.cc:2454] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf278f11f4dc463b985864225944d329 in term 0.
I20260924 19:36:34.687248 2025 raft_consensus.cc:504] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.687368 2025 leader_election.cc:288] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [CANDIDATE]: Term 1 election: Requested vote from peers 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123), a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683)
I20260924 19:36:34.687527 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15bdd47137a241998e139135103705b1" candidate_uuid: "bf278f11f4dc463b985864225944d329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1236539bcac74c49978ad2bbcb535615"
I20260924 19:36:34.687597 1665 raft_consensus.cc:3037] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.688275 1665 raft_consensus.cc:2454] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf278f11f4dc463b985864225944d329 in term 1.
I20260924 19:36:34.688371 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15bdd47137a241998e139135103705b1" candidate_uuid: "bf278f11f4dc463b985864225944d329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55e7b49240546f9bf83763b0f43230f"
I20260924 19:36:34.688427 1738 leader_election.cc:302] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1236539bcac74c49978ad2bbcb535615, bf278f11f4dc463b985864225944d329; no voters:
I20260924 19:36:34.688444 1947 raft_consensus.cc:3037] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:34.688499 2025 raft_consensus.cc:2781] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:34.688622 2025 raft_consensus.cc:686] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [term 1 LEADER]: Becoming Leader. State: Replica: bf278f11f4dc463b985864225944d329, State: Running, Role: LEADER
I20260924 19:36:34.688684 2025 consensus_queue.cc:237] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260924 19:36:34.689036 2061 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: TxnStatusTablet state changed. Reason: New leader bf278f11f4dc463b985864225944d329. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.689105 2061 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:34.689203 1947 raft_consensus.cc:2454] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf278f11f4dc463b985864225944d329 in term 1.
I20260924 19:36:34.689363 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:36:34.689376 1506 catalog_manager.cc:5495] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf278f11f4dc463b985864225944d329 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:34.689468 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:36:34.703208 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.700670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.0.98.2:46341)
0924 19:36:34.700719 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.700721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.700730 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:34.700817 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.700818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.700833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.700884 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:34.700993 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.701474 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.702543 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.702720 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.702763 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.702933 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.703025 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.703072 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.703128 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.703145 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:36:34.703256 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.700605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46341 (local address 127.0.0.1:49602)
0924 19:36:34.700681 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.700693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.700741 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.700888 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.700889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.700890 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:34.700943 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.700945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.701532 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.701533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.702473 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.702476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.702774 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.702774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.702839 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.702857 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.703074 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.703078 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.703125 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.703175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:36:34.704622 1665 raft_consensus.cc:1261] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Refusing update from remote peer bf278f11f4dc463b985864225944d329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.704622 1947 raft_consensus.cc:1261] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Refusing update from remote peer bf278f11f4dc463b985864225944d329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.704870 2025 consensus_queue.cc:1035] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.705003 2061 consensus_queue.cc:1035] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.705601 2048 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: TxnStatusTablet state changed. Reason: New leader bf278f11f4dc463b985864225944d329. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.705699 2048 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.705729 2037 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: TxnStatusTablet state changed. Reason: New leader bf278f11f4dc463b985864225944d329. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.705804 2037 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.706377 2061 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.706453 2061 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:34.706537 2061 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.706575 2061 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P bf278f11f4dc463b985864225944d329: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:34.706831 2048 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.706904 2048 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P a55e7b49240546f9bf83763b0f43230f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.707321 2031 mvcc.cc:205] Tried to move back new op lower bound from 7332981123908136960 to 7332981123845718016. Current Snapshot: MvccSnapshot[applied={T|T < 7332981123908136960 or (T in {7332981123908136960})}]
I20260924 19:36:34.707423 2037 tablet_replica.cc:439] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf278f11f4dc463b985864225944d329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 } } peers { permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 } } peers { permanent_uuid: "a55e7b49240546f9bf83763b0f43230f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260924 19:36:34.707485 2037 tablet_replica.cc:442] T 15bdd47137a241998e139135103705b1 P 1236539bcac74c49978ad2bbcb535615: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:34.710475 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.708000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.0.98.3:36683)
0924 19:36:34.708054 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.708057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.708069 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:36:34.708115 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.708116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.708144 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.708250 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:34.708325 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.709053 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.709750 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.710046 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.710109 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.710192 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.710309 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.710374 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.710388 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.710406 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:34.715180 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.712894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41341 (local address 127.0.98.2:46341)
0924 19:36:34.712944 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.712946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.712952 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:36:34.712969 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.712970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.712977 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.713017 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:34.713145 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.713579 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.714613 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.714786 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.714828 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.715004 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.715027 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.715055 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.715109 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.715124 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:36:34.715214 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.712698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46341 (local address 127.0.98.3:41341)
0924 19:36:34.712759 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.712770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.712812 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.713036 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.713036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.713038 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:34.713089 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.713091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.713635 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.713636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.714537 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.714540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.714841 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.714842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.714906 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.714927 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.715058 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.715059 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.715104 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.715143 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:36:34.718050 1665 raft_consensus.cc:1261] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Refusing update from remote peer a55e7b49240546f9bf83763b0f43230f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.718050 1806 raft_consensus.cc:1261] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Refusing update from remote peer a55e7b49240546f9bf83763b0f43230f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:34.718259 2021 consensus_queue.cc:1035] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1236539bcac74c49978ad2bbcb535615" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.718381 2048 consensus_queue.cc:1035] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf278f11f4dc463b985864225944d329" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:34.726740 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.724235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44031 (local address 127.0.98.3:36683)
0924 19:36:34.724272 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.724274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.724409 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:36:34.724455 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.724456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.724465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.724508 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:34.724636 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.725104 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.726035 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.726281 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.726312 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.726498 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.726532 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.726577 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.726635 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.726655 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:36:34.726761 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.724202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.2:44031)
0924 19:36:34.724350 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:34.724360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:34.724411 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:34.724509 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:34.724509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:34.724521 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:34.724585 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.724586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.725167 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.725168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.725953 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:34.725955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.726372 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.726372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:34.726415 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.726428 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:34.726584 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:34.726585 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:34.726631 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:36:34.726672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 19:36:34.741837 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:34.741952 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:34.743037 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 079913ac08d542438479a678ac05a547: Sending DeleteTablet for 3 replicas of tablet 17c629b7a476426f94db838ce7c06203
I20260924 19:36:34.743456 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c629b7a476426f94db838ce7c06203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34010
I20260924 19:36:34.743562 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c629b7a476426f94db838ce7c06203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33592
I20260924 19:36:34.743659 2043 tablet_replica.cc:330] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: stopping tablet replica
I20260924 19:36:34.743662 2042 tablet_replica.cc:330] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: stopping tablet replica
I20260924 19:36:34.743723 2043 raft_consensus.cc:2229] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:34.743734 2042 raft_consensus.cc:2229] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:34.743767 2043 raft_consensus.cc:2258] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.743831 2042 raft_consensus.cc:2258] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.744144 2043 ts_tablet_manager.cc:1905] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.744143 2042 ts_tablet_manager.cc:1905] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.744207 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c629b7a476426f94db838ce7c06203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49590
I20260924 19:36:34.744297 2041 tablet_replica.cc:330] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: stopping tablet replica
I20260924 19:36:34.744336 2041 raft_consensus.cc:2229] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:34.744372 2041 raft_consensus.cc:2258] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:34.745734 2043 ts_tablet_manager.cc:1918] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:34.745734 2042 ts_tablet_manager.cc:1918] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:34.745790 2043 log.cc:1196] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/17c629b7a476426f94db838ce7c06203
I20260924 19:36:34.745790 2042 log.cc:1196] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/17c629b7a476426f94db838ce7c06203
I20260924 19:36:34.745981 2043 ts_tablet_manager.cc:1939] T 17c629b7a476426f94db838ce7c06203 P 1236539bcac74c49978ad2bbcb535615: Deleting consensus metadata
I20260924 19:36:34.746416 1493 catalog_manager.cc:4841] TS 1236539bcac74c49978ad2bbcb535615 (127.0.98.1:34123): tablet 17c629b7a476426f94db838ce7c06203 (table test-table [id=896d083b6f0c4ccba8351ae50c0d1342]) successfully deleted
I20260924 19:36:34.748231 2042 ts_tablet_manager.cc:1939] T 17c629b7a476426f94db838ce7c06203 P a55e7b49240546f9bf83763b0f43230f: Deleting consensus metadata
I20260924 19:36:34.748334 2041 ts_tablet_manager.cc:1905] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:34.749853 2041 ts_tablet_manager.cc:1918] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:34.749902 2041 log.cc:1196] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/17c629b7a476426f94db838ce7c06203
I20260924 19:36:34.750082 2041 ts_tablet_manager.cc:1939] T 17c629b7a476426f94db838ce7c06203 P bf278f11f4dc463b985864225944d329: Deleting consensus metadata
I20260924 19:36:34.750289 1493 catalog_manager.cc:4841] TS a55e7b49240546f9bf83763b0f43230f (127.0.98.3:36683): tablet 17c629b7a476426f94db838ce7c06203 (table test-table [id=896d083b6f0c4ccba8351ae50c0d1342]) successfully deleted
I20260924 19:36:34.750591 1492 catalog_manager.cc:4841] TS bf278f11f4dc463b985864225944d329 (127.0.98.2:46341): tablet 17c629b7a476426f94db838ce7c06203 (table test-table [id=896d083b6f0c4ccba8351ae50c0d1342]) successfully deleted
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278594, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:34.764137 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.753866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.0.98.1:34123)
0924 19:36:34.753913 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.753914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.754016 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:36:34.754107 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.754108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.754118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.754147 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.754890 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.755636 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.756825 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.757102 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.757169 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.760757 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.760770 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.760770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.760778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.762804 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.763196 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.763200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.763200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.763259 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.763446 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.763447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.763447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.763606 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.763726 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.763958 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.764039 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:36:34.764899 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34036
I20260924 19:36:34.773886 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:34.766382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.98.62:34663)
0924 19:36:34.766426 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:34.766428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:34.766495 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:36:34.766589 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:34.766590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:34.766601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:34.766629 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:34.767355 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.768205 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.769267 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:34.769568 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:34.769606 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:34.771000 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:34.771001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:34.771001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:34.771009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:34.772701 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.773073 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.773074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.773075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.773135 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:34.773316 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:34.773317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:34.773317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:34.773433 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:34.773507 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:34.773712 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:36:34.773786 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 19:36:34.774654 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39012
I20260924 19:36:34.775292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1579
I20260924 19:36:34.785209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1720
I20260924 19:36:34.790091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1861
I20260924 19:36:34.794076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 1475
2026-09-24T19:36:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6629 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[2084](info): set up 2 sockets
Sep 24 19:36:34 dist-test-slave-6kf3 krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260924 19:36:35.739558 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:35 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:35.846802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45617
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:35.924810 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:35.925028 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:35.925051 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:35.926198 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:35.926240 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:36:35.926252 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:35.926265 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:35.926275 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:35.926286 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:35.927804 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45617
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:35.928076 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:35.928287 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:35.932794 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:35.932842 2097 server_base.cc:1034] running on GCE node
W20260924 19:36:35.932818 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:35.932796 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:35.933295 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:35.933580 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:35.934710 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790278595934692 us; error 33 us; skew 500 ppm
Sep 24 19:36:35 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:35.936993 2097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:35.937386 2097 webserver.cc:460] Webserver started at http://127.0.98.62:46479/ using document root <none> and password file <none>
I20260924 19:36:35.937579 2097 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:35.937618 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:35.937707 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:35.938699 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "695d943e761b4a6f9107baac333cc442"
format_stamp: "Formatted at 2026-09-24 19:36:35 on dist-test-slave-6kf3"
server_key: "2bb32155c47d01fcb109c10884536865"
server_key_iv: "3f058edb06f1d14b6919fe558837e73c"
server_key_version: "encryptionkey@0"
I20260924 19:36:35.939002 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "695d943e761b4a6f9107baac333cc442"
format_stamp: "Formatted at 2026-09-24 19:36:35 on dist-test-slave-6kf3"
server_key: "2bb32155c47d01fcb109c10884536865"
server_key_iv: "3f058edb06f1d14b6919fe558837e73c"
server_key_version: "encryptionkey@0"
I20260924 19:36:35.940145 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:35.940765 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:35.940887 2097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:35.940975 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "695d943e761b4a6f9107baac333cc442"
format_stamp: "Formatted at 2026-09-24 19:36:35 on dist-test-slave-6kf3"
server_key: "2bb32155c47d01fcb109c10884536865"
server_key_iv: "3f058edb06f1d14b6919fe558837e73c"
server_key_version: "encryptionkey@0"
I20260924 19:36:35.941035 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:35.960620 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:35.962356 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:35.962476 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:35.966745 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45617
I20260924 19:36:35.966792 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45617 every 8 connection(s)
I20260924 19:36:35.967101 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:35.967689 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:35.969874 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Bootstrap starting.
I20260924 19:36:35.970420 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:35.970674 2175 log.cc:824] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:35.971004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2097
I20260924 19:36:35.971079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:35.971235 392 external_mini_cluster.cc:1428] Setting key 01990b7fee572bd69b23eb22ae79424f
I20260924 19:36:35.971385 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: No bootstrap required, opened a new log
I20260924 19:36:35.973671 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } }
I20260924 19:36:35.973802 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:35.973824 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695d943e761b4a6f9107baac333cc442, State: Initialized, Role: FOLLOWER
I20260924 19:36:35.973929 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } }
I20260924 19:36:35.974007 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:35.974044 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:35.974087 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:35.974602 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } }
I20260924 19:36:35.974689 2175 leader_election.cc:302] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695d943e761b4a6f9107baac333cc442; no voters:
I20260924 19:36:35.974831 2175 leader_election.cc:288] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:35.974855 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:35.975009 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [term 1 LEADER]: Becoming Leader. State: Replica: 695d943e761b4a6f9107baac333cc442, State: Running, Role: LEADER
I20260924 19:36:35.975075 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:35.975142 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } }
I20260924 19:36:35.975492 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695d943e761b4a6f9107baac333cc442. Latest consensus state: current_term: 1 leader_uuid: "695d943e761b4a6f9107baac333cc442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } } }
I20260924 19:36:35.975641 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:35.976385 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695d943e761b4a6f9107baac333cc442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d943e761b4a6f9107baac333cc442" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45617 } } }
I20260924 19:36:35.976610 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442 [sys.catalog]: This master's current role is: LEADER
W20260924 19:36:35.977527 2196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:35.977571 2196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:35.977609 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:35.977744 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:36:35 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:35.979538 2189 catalog_manager.cc:1347] Generated new cluster ID: ac5c250e564946f497cde57176c034b5
I20260924 19:36:35.979588 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:35.981963 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:35.971696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.0.98.62:45617)
0924 19:36:35.971869 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:35.971879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:35.971906 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:36:35.972604 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:35.972610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:35.972657 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:35.972784 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:35.973363 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:35.974373 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:35.975305 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:35.975832 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:35.975957 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:35.978400 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:35.978404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:35.978404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:35.978415 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:35.980219 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:35.980550 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:35.980553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:35.980553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:35.980641 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:35.980792 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:35.980793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:35.980793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:35.981057 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:35.981187 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:35.981637 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 19:36:35.981760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260924 19:36:35.994345 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:35.994875 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:36.000550 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Generated new TSK 0
I20260924 19:36:36.000716 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:36.025981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45617
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:36.103822 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:36.104043 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:36.104065 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:36.105168 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:36.105208 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:36.105226 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:36:36.105238 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:36.105268 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:36.106700 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:36.106921 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:36.107131 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:36.111703 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:36.111697 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:36.111724 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:36.112025 2201 server_base.cc:1034] running on GCE node
I20260924 19:36:36.112191 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:36.112455 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:36.113593 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790278596113579 us; error 26 us; skew 500 ppm
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:36.115903 2201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:36.116328 2201 webserver.cc:460] Webserver started at http://127.0.98.1:38443/ using document root <none> and password file <none>
I20260924 19:36:36.116539 2201 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:36.116585 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:36.116705 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:36.117719 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "9f7e7ae3b38f935a4c33d67e379a91fc"
server_key_iv: "32c99ebec2b33dae808d20d7a4ca37af"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.118041 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "9f7e7ae3b38f935a4c33d67e379a91fc"
server_key_iv: "32c99ebec2b33dae808d20d7a4ca37af"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.119230 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:36.119880 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.120041 2201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.120113 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "9f7e7ae3b38f935a4c33d67e379a91fc"
server_key_iv: "32c99ebec2b33dae808d20d7a4ca37af"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.120164 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:36.127035 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:36.129004 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:36.129107 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:36.129635 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:36.129671 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.129700 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:36.129724 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:36.136154 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40963
I20260924 19:36:36.136221 2331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40963 every 8 connection(s)
I20260924 19:36:36.136552 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:36.138937 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.129959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50605 (local address 127.0.98.62:45617)
0924 19:36:36.130002 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.130004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.130247 (+ 243us) server_negotiation.cc:407] Connection header received
0924 19:36:36.130711 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.130712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.130725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.130798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.131515 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.132309 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.133316 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.133624 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.133794 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.136391 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.136392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.136392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.136399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.137724 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.138079 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.138080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.138080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.138138 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.138343 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.138344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.138344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.138461 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.138514 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.138761 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.138846 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:36.139096 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.129962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.1:50605)
0924 19:36:36.130122 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.130156 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.130627 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.130846 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.130850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.131144 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.131419 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.131425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.132397 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.132398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.133233 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.133236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.133813 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.133814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.133985 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.134524 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.134534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.136307 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.137796 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.137802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.137804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.138013 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.138189 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.138190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.138190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.138291 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.138512 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.138518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.138649 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.138735 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.138882 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260924 19:36:36.139564 2332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45617
I20260924 19:36:36.139640 2332 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:36.139796 2332 heartbeater.cc:511] Master 127.0.98.62:45617 requested a full tablet report, sending...
I20260924 19:36:36.140107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2201
I20260924 19:36:36.140210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:36.140343 2128 ts_manager.cc:194] Registered new tserver with Master: 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963)
I20260924 19:36:36.140374 392 external_mini_cluster.cc:1428] Setting key b55450c999a5b9706619fc541db0bbd6
I20260924 19:36:36.141129 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50605
I20260924 19:36:36.146266 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.140604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.1:45273)
0924 19:36:36.140729 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.140738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.140792 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.140895 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.140895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.141054 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.141109 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.141110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.141833 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.141834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.142545 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.142548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.142877 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.142877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.142958 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.143411 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.143417 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.143740 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.145289 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.145290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.145290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.145446 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.145643 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.145644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.145644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.145708 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.146015 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.146016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.146054 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.146101 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.146185 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260924 19:36:36.146289 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.140562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45273 (local address 127.0.98.62:45617)
0924 19:36:36.140601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.140602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.140779 (+ 177us) server_negotiation.cc:407] Connection header received
0924 19:36:36.140833 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.140833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.140842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.140883 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.141169 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.141752 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.142633 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.142821 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.142867 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.143803 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.143805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.143805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.143812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.145217 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.145507 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.145508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.145508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.145565 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.145819 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.145820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.145820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.145960 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.146026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.146128 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.146201 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:36.162401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45617
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:36.240548 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:36.240753 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:36.240777 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:36.241904 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:36.241945 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:36.241964 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:36:36.241974 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:36.241994 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:36.243427 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:36.243635 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:36.243850 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:36.248649 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:36.248718 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:36.248805 2341 server_base.cc:1034] running on GCE node
W20260924 19:36:36.248718 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:36.249091 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:36.249301 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:36.250427 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790278596250417 us; error 28 us; skew 500 ppm
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:36.252701 2341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:36.253098 2341 webserver.cc:460] Webserver started at http://127.0.98.2:46035/ using document root <none> and password file <none>
I20260924 19:36:36.253288 2341 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:36.253331 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:36.253458 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:36.254442 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffa91c44742f47bca2bb696f21777c7c"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "fcb0a3ad4011a7270cf8eb97dfb20dfa"
server_key_iv: "6d239e904400bbc105efcb44d2d16f25"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.254755 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffa91c44742f47bca2bb696f21777c7c"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "fcb0a3ad4011a7270cf8eb97dfb20dfa"
server_key_iv: "6d239e904400bbc105efcb44d2d16f25"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.255897 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:36.256539 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.256708 2341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.256790 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "ffa91c44742f47bca2bb696f21777c7c"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "fcb0a3ad4011a7270cf8eb97dfb20dfa"
server_key_iv: "6d239e904400bbc105efcb44d2d16f25"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.256839 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:36.276233 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:36.278298 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:36.278409 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:36.278851 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:36.278898 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.278919 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:36.278930 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.284914 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32819
I20260924 19:36:36.285025 2471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32819 every 8 connection(s)
I20260924 19:36:36.285257 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:36.286598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2341
I20260924 19:36:36.286675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:36.286828 392 external_mini_cluster.cc:1428] Setting key d69a89876a3b8d0d26d2c1bdf59827d0
I20260924 19:36:36.288339 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.279308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50897 (local address 127.0.98.62:45617)
0924 19:36:36.279352 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.279354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.279607 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:36:36.280135 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.280137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.280150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.280201 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.280989 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.281761 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.283247 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.283545 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.283626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.285840 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.285841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.285842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.285848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.287126 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.287490 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.287491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.287491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.287551 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.287783 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.287783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.287783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.287874 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.287919 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.288222 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.288279 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:36.288553 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.279318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.2:50897)
0924 19:36:36.279493 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.279528 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.280039 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.280272 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.280276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.280609 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.280888 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.280895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.281862 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.281863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.283122 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.283124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.283625 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.283626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.283731 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.284266 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.284276 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.285764 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.287192 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.287197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.287199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.287429 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.287602 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.287603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.287603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.287731 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.287922 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.287926 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.288100 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.288199 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.288349 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260924 19:36:36.289000 2472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45617
I20260924 19:36:36.289076 2472 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:36.289237 2472 heartbeater.cc:511] Master 127.0.98.62:45617 requested a full tablet report, sending...
I20260924 19:36:36.289532 2129 ts_manager.cc:194] Registered new tserver with Master: ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819)
I20260924 19:36:36.289942 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50897
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:36:36.294905 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.289952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.2:54325)
0924 19:36:36.290244 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.290252 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.290305 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.290419 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.290420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.290576 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.290622 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.290623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.291234 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.291235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.291736 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.291737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.292101 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.292102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.292168 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.292537 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.292540 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.292789 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.294040 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.294041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.294041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.294191 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.294346 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.294347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.294347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.294411 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.294641 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.294642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.294682 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.294747 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.294832 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260924 19:36:36.294955 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.290003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54325 (local address 127.0.98.62:45617)
0924 19:36:36.290060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.290062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.290301 (+ 239us) server_negotiation.cc:407] Connection header received
0924 19:36:36.290358 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.290358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.290368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.290407 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.290679 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.291165 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.291798 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.292040 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.292085 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.292858 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.292859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.292859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.292865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.293973 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.294246 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.294247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.294248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.294297 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.294457 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.294458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.294458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.294588 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.294643 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.294784 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.294863 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:36.310528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45617
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:36.388190 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:36.388388 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:36.388410 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:36.389550 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:36.389587 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:36.389606 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:36:36.389619 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:36:36.389636 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:36.391039 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:36.391223 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:36.391407 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:36.396026 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:36.396006 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:36.396003 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:36.396075 2481 server_base.cc:1034] running on GCE node
I20260924 19:36:36.396360 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:36.396562 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:36.397689 2481 hybrid_clock.cc:648] HybridClock initialized: now 1790278596397655 us; error 49 us; skew 500 ppm
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:36.399820 2481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:36.400223 2481 webserver.cc:460] Webserver started at http://127.0.98.3:43383/ using document root <none> and password file <none>
I20260924 19:36:36.400416 2481 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:36.400456 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:36.400538 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:36.401494 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "e1266c767579698a5d7d5e85a060b4ba"
server_key_iv: "c1da9c2691eef4f4616399b798427b1b"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.401769 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "e1266c767579698a5d7d5e85a060b4ba"
server_key_iv: "c1da9c2691eef4f4616399b798427b1b"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.402946 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:36:36.403599 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.403738 2481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:36.403815 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
format_stamp: "Formatted at 2026-09-24 19:36:36 on dist-test-slave-6kf3"
server_key: "e1266c767579698a5d7d5e85a060b4ba"
server_key_iv: "c1da9c2691eef4f4616399b798427b1b"
server_key_version: "encryptionkey@0"
I20260924 19:36:36.403872 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:36.420037 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:36.421981 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:36.422107 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:36.422585 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:36.422631 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.422659 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:36.422680 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.428795 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35323
I20260924 19:36:36.428920 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35323 every 8 connection(s)
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:36.429164 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:36.431668 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.422998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59775 (local address 127.0.98.62:45617)
0924 19:36:36.423054 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.423057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.423307 (+ 250us) server_negotiation.cc:407] Connection header received
0924 19:36:36.423780 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.423781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.423793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.423880 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.424623 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.425397 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.426862 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.427142 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.427215 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.429613 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.429614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.429614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.429622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.430653 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.431000 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.431001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.431001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.431045 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.431219 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.431220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.431220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.431311 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.431354 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.431551 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.431607 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:36:36.431965 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.422995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.3:59775)
0924 19:36:36.423175 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.423216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.423681 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.423964 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.423969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.424270 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.424533 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.424542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.425483 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.425484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.426759 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.426763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.427216 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.427216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.427316 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.427807 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.427817 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.429541 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.430728 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.430734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.430735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.430946 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.431087 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.431087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.431087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.431179 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.431351 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.431353 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.431459 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.431532 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.431653 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260924 19:36:36.432421 2612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45617
I20260924 19:36:36.432495 2612 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:36.432667 2612 heartbeater.cc:511] Master 127.0.98.62:45617 requested a full tablet report, sending...
I20260924 19:36:36.433014 2128 ts_manager.cc:194] Registered new tserver with Master: 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:36.433423 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59775
I20260924 19:36:36.434767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2481
I20260924 19:36:36.434841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:36.435052 392 external_mini_cluster.cc:1428] Setting key cb0c465c5f5343a0775774af8a4a9e90
I20260924 19:36:36.435866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:36:36.438947 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.433720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56091 (local address 127.0.98.62:45617)
0924 19:36:36.433759 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.433761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.433880 (+ 119us) server_negotiation.cc:407] Connection header received
0924 19:36:36.433932 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.433933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.433940 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.433981 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.434273 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.434798 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.435703 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.436010 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.436068 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.436856 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.436857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.436857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.436863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.438017 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.438383 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.438383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.438384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.438424 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.438581 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.438582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.438582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.438663 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.438703 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.438829 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.438892 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:36:36.439005 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.433645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.98.3:56091)
0924 19:36:36.433808 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.433818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.433881 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.433986 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.433986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.434163 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.434218 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.434219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.434932 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.434933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.435644 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.435646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.436068 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.436069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.436125 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.436499 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.436503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.436773 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.438090 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.438091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.438091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.438322 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.438474 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.438475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.438475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.438533 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.438706 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.438707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.438748 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.438814 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.438912 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:36.458163 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.451379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.0.98.62:45617)
0924 19:36:36.451432 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.451434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.451442 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:36.451532 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.451533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.451541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.451585 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.452353 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.452853 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.453581 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.453767 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.453819 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.455875 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.455877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.455877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.455884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.457167 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.457508 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.457509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.457509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.457570 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.457796 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.457796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.457797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.457898 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.457971 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.458041 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.458099 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:36:36.459697 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:36.461345 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:36.467409 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35323 (local address 127.0.0.1:55282)
0924 19:36:36.463431 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.463474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.463536 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.463891 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.463894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.463897 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.464038 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.464040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.465401 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.465402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.466223 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.466225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.467110 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.467111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.467193 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.467208 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.467277 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.467336 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260924 19:36:36.467459 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.98.1:40963)
0924 19:36:36.463843 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.463847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.463855 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:36.463892 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.463894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.463908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.464027 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.464197 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.465259 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.466436 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.467090 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.467146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.467192 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.467319 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.467368 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260924 19:36:36.467661 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.0.1:34114)
0924 19:36:36.463679 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.463687 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.463734 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.464091 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.464091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.464093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.464126 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.464127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.465342 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.465343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.466376 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.466378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.467155 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.467155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.467202 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.467209 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.467282 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.467622 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":307,"thread_start_us":58,"threads_started":1}
I20260924 19:36:36.467439 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.0.98.3:35323)
0924 19:36:36.463732 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.463738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.463748 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:36.463786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.463787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.463802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.463885 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.464115 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.465298 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.466292 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.467013 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.467112 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.467181 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.467311 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.467350 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260924 19:36:36.468274 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.0.1:32972)
0924 19:36:36.463612 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.463621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.463670 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.464066 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.464067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.464069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.464126 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.464127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.465009 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.465010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.466219 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.466221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.468063 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.468064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.468124 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.468128 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.468173 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.468207 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260924 19:36:36.468357 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec519d7fef34599a0f1a82d02239176 (DEFAULT_TABLE table=test-table [id=0c43b28e3d214100ad43c679646617cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.468348 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec519d7fef34599a0f1a82d02239176 (DEFAULT_TABLE table=test-table [id=0c43b28e3d214100ad43c679646617cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.468688 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec519d7fef34599a0f1a82d02239176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.468693 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec519d7fef34599a0f1a82d02239176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.468713 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.463534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.0.98.2:32819)
0924 19:36:36.463889 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.463893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.463902 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:36.463954 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.463956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.463970 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.464197 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.464253 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.464937 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.467218 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.467975 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.468038 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.468088 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.468585 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.468624 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":82,"threads_started":1}
I20260924 19:36:36.471225 2632 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Bootstrap starting.
I20260924 19:36:36.471357 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec519d7fef34599a0f1a82d02239176 (DEFAULT_TABLE table=test-table [id=0c43b28e3d214100ad43c679646617cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.471513 2633 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Bootstrap starting.
I20260924 19:36:36.471671 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec519d7fef34599a0f1a82d02239176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.472121 2632 tablet_bootstrap.cc:654] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.472126 2633 tablet_bootstrap.cc:654] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.472385 2632 log.cc:824] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:36.472409 2633 log.cc:824] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:36.473027 2632 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: No bootstrap required, opened a new log
I20260924 19:36:36.473088 2632 ts_tablet_manager.cc:1397] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:36.473233 2633 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: No bootstrap required, opened a new log
I20260924 19:36:36.473304 2633 ts_tablet_manager.cc:1397] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:36.474633 2632 raft_consensus.cc:348] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.474772 2632 raft_consensus.cc:374] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.474802 2632 raft_consensus.cc:729] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.475000 2632 consensus_queue.cc:260] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.475158 2633 raft_consensus.cc:348] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.475276 2633 raft_consensus.cc:374] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.475274 2632 ts_tablet_manager.cc:1428] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:36.475311 2633 raft_consensus.cc:729] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.475348 2612 heartbeater.cc:503] Master 127.0.98.62:45617 was elected leader, sending a full tablet report...
I20260924 19:36:36.475438 2633 consensus_queue.cc:260] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.475503 2636 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Bootstrap starting.
I20260924 19:36:36.475694 2332 heartbeater.cc:503] Master 127.0.98.62:45617 was elected leader, sending a full tablet report...
I20260924 19:36:36.475966 2633 ts_tablet_manager.cc:1428] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:36.476163 2636 tablet_bootstrap.cc:654] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.476516 2636 log.cc:824] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:36.477118 2636 tablet_bootstrap.cc:492] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: No bootstrap required, opened a new log
I20260924 19:36:36.477167 2636 ts_tablet_manager.cc:1397] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:36.478462 2636 raft_consensus.cc:348] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.478621 2636 raft_consensus.cc:374] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.478667 2636 raft_consensus.cc:729] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.478782 2636 consensus_queue.cc:260] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.478993 2636 ts_tablet_manager.cc:1428] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:36.479048 2472 heartbeater.cc:503] Master 127.0.98.62:45617 was elected leader, sending a full tablet report...
W20260924 19:36:36.536070 2473 tablet.cc:2369] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:36:36.637157 2333 tablet.cc:2369] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:36:36.679682 2613 tablet.cc:2369] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:36:36.794708 2640 raft_consensus.cc:482] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:36.794824 2640 raft_consensus.cc:504] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.795190 2640 leader_election.cc:288] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323), 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963)
I20260924 19:36:36.797950 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.795497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58665 (local address 127.0.98.3:35323)
0924 19:36:36.795553 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.795555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.795565 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:36.795698 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.795699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.795708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.795751 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.795885 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.796477 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797311 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.797665 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.797731 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.797851 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.797886 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:36:36.797959 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.795649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36865 (local address 127.0.98.1:40963)
0924 19:36:36.795710 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.795712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.795721 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:36.795748 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.795749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.795759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.795804 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.795957 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.796406 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797321 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.797668 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797723 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.797732 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.797863 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.797898 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:36:36.797983 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.795559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.98.2:36865)
0924 19:36:36.795639 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.795649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.795700 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.795829 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.795830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.795832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.795888 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.795889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.796473 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.796474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.797233 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.797235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.797732 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.797804 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.797807 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.797850 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.797890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1,"thread_start_us":34,"threads_started":1}
I20260924 19:36:36.797986 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.795395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35323 (local address 127.0.98.2:58665)
0924 19:36:36.795493 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.795503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.795572 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.795768 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.795768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.795771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.795832 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.795834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.796554 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.796554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.797239 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.797240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.797729 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.797730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.797794 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.797798 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.797841 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.797881 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260924 19:36:36.798336 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec519d7fef34599a0f1a82d02239176" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" is_pre_election: true
I20260924 19:36:36.798345 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec519d7fef34599a0f1a82d02239176" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" is_pre_election: true
I20260924 19:36:36.798463 2286 raft_consensus.cc:2454] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:36.798463 2566 raft_consensus.cc:2454] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:36.798660 2357 leader_election.cc:302] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc8d3e1dd2d47b1abb8194d10c3924f, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:36.798775 2640 raft_consensus.cc:2781] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:36.798837 2640 raft_consensus.cc:482] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:36.798851 2640 raft_consensus.cc:3037] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.799474 2640 raft_consensus.cc:504] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.799572 2640 leader_election.cc:288] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Requested vote from peers 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323), 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963)
I20260924 19:36:36.799701 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec519d7fef34599a0f1a82d02239176" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
I20260924 19:36:36.799759 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec519d7fef34599a0f1a82d02239176" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
I20260924 19:36:36.799772 2566 raft_consensus.cc:3037] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.799829 2286 raft_consensus.cc:3037] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.800457 2286 raft_consensus.cc:2454] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:36.800464 2566 raft_consensus.cc:2454] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:36.800582 2357 leader_election.cc:302] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc8d3e1dd2d47b1abb8194d10c3924f, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:36.800678 2640 raft_consensus.cc:2781] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:36.800781 2640 raft_consensus.cc:686] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 1 LEADER]: Becoming Leader. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Running, Role: LEADER
I20260924 19:36:36.800875 2640 consensus_queue.cc:237] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260924 19:36:36.801622 2128 catalog_manager.cc:5495] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c reported cstate change: term changed from 0 to 1, leader changed from <none> to ffa91c44742f47bca2bb696f21777c7c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: HEALTHY } } }
Sep 24 19:36:36 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:36.847517 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.832631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.0.98.62:45617)
0924 19:36:36.832685 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.832687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.832818 (+ 131us) server_negotiation.cc:407] Connection header received
0924 19:36:36.832886 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.832886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.832896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.832945 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:36.833377 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.833871 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.834530 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.834707 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.834757 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.845454 (+ 10697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:36.845456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:36.845457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:36.845464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:36.846516 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.846879 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.846879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.846880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.846923 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:36.847100 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:36.847100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:36.847101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:36.847207 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:36.847248 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.847381 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.847451 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:36:36.847615 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.832561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45617 (local address 127.0.0.1:53340)
0924 19:36:36.832736 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.832745 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.832828 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.832952 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.832953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.833261 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:36.833310 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.833312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.833934 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.833935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.834465 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.834467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.834759 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.834760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.834818 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.843807 (+ 8989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:36.843813 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:36.845356 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:36.846596 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.846605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.846607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.846818 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.846987 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:36.846987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:36.846988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:36.847052 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:36.847246 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:36.847253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:36.847313 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.847361 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.847514 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":56,"thread_start_us":91,"threads_started":1}
I20260924 19:36:36.849295 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:36:36.849486 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:36.850871 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 23dab5a2ca5146eaaea7ed23385a3ece (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583e17d0fd834f50b1acac93d0abd527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:36.850870 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 23dab5a2ca5146eaaea7ed23385a3ece (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583e17d0fd834f50b1acac93d0abd527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:36.851013 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dab5a2ca5146eaaea7ed23385a3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.851016 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dab5a2ca5146eaaea7ed23385a3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.852061 2633 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: Bootstrap starting.
I20260924 19:36:36.852125 2632 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: Bootstrap starting.
I20260924 19:36:36.852185 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 23dab5a2ca5146eaaea7ed23385a3ece (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583e17d0fd834f50b1acac93d0abd527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:36:36.852339 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dab5a2ca5146eaaea7ed23385a3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.852444 2633 tablet_bootstrap.cc:654] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.852548 2632 tablet_bootstrap.cc:654] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.853076 2632 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: No bootstrap required, opened a new log
I20260924 19:36:36.853116 2632 ts_tablet_manager.cc:1397] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:36.853247 2632 raft_consensus.cc:348] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.853327 2632 raft_consensus.cc:374] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.853344 2632 raft_consensus.cc:729] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.853381 2632 consensus_queue.cc:260] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.853467 2632 ts_tablet_manager.cc:1428] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.853538 2637 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.853607 2637 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.853621 2636 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: Bootstrap starting.
I20260924 19:36:36.854003 2636 tablet_bootstrap.cc:654] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.854085 2633 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: No bootstrap required, opened a new log
I20260924 19:36:36.854117 2633 ts_tablet_manager.cc:1397] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:36.854233 2633 raft_consensus.cc:348] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.854275 2633 raft_consensus.cc:374] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.854295 2633 raft_consensus.cc:729] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.854326 2633 consensus_queue.cc:260] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.854393 2633 ts_tablet_manager.cc:1428] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.854532 2638 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.854617 2638 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.854777 2636 tablet_bootstrap.cc:492] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: No bootstrap required, opened a new log
I20260924 19:36:36.854815 2636 ts_tablet_manager.cc:1397] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:36.854943 2636 raft_consensus.cc:348] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.855003 2636 raft_consensus.cc:374] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.855020 2636 raft_consensus.cc:729] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.855062 2636 consensus_queue.cc:260] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.855145 2636 ts_tablet_manager.cc:1428] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.855192 2640 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.855276 2640 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.873081 2640 raft_consensus.cc:482] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:36.873165 2640 raft_consensus.cc:504] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.873301 2640 leader_election.cc:288] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:36.873472 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dab5a2ca5146eaaea7ed23385a3ece" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" is_pre_election: true
I20260924 19:36:36.873499 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dab5a2ca5146eaaea7ed23385a3ece" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" is_pre_election: true
I20260924 19:36:36.873555 2286 raft_consensus.cc:2454] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:36.873588 2566 raft_consensus.cc:2454] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:36.873696 2357 leader_election.cc:302] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fbfd39d6ff4cb4bdc919ab049b42bb, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:36.873811 2640 raft_consensus.cc:2781] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:36.873852 2640 raft_consensus.cc:482] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:36.873872 2640 raft_consensus.cc:3037] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.874521 2640 raft_consensus.cc:504] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.874634 2640 leader_election.cc:288] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Requested vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:36.874755 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dab5a2ca5146eaaea7ed23385a3ece" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
I20260924 19:36:36.874828 2286 raft_consensus.cc:3037] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.874809 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dab5a2ca5146eaaea7ed23385a3ece" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
I20260924 19:36:36.874886 2566 raft_consensus.cc:3037] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.875458 2286 raft_consensus.cc:2454] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:36.875593 2566 raft_consensus.cc:2454] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:36.875630 2357 leader_election.cc:302] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fbfd39d6ff4cb4bdc919ab049b42bb, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:36.875725 2640 raft_consensus.cc:2781] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:36.875782 2640 raft_consensus.cc:686] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [term 1 LEADER]: Becoming Leader. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Running, Role: LEADER
I20260924 19:36:36.875828 2640 consensus_queue.cc:237] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.876132 2643 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: TxnStatusTablet state changed. Reason: New leader ffa91c44742f47bca2bb696f21777c7c. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.876227 2643 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:36.876449 2655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:36:36.876402 2128 catalog_manager.cc:5495] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c reported cstate change: term changed from 0 to 1, leader changed from <none> to ffa91c44742f47bca2bb696f21777c7c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:36.876576 2655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:36:36.885422 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.882760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.0.1:32980)
0924 19:36:36.882825 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.882837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.882884 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.883039 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.883040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.883041 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:36.883081 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.883084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.883972 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.883974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.884548 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.884550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.884988 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.884988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.885040 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.885052 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.885299 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.885303 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.885326 (+ 23us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.885367 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:36:36.885424 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.882820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.98.2:32819)
0924 19:36:36.882865 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.882867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.882878 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:36:36.882945 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.882946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.882967 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.883040 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.883119 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.883854 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.884626 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.884915 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.884981 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.885130 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.885244 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.885318 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.885334 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.885355 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:36.887223 2566 raft_consensus.cc:1261] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.887223 2286 raft_consensus.cc:1261] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.887532 2640 consensus_queue.cc:1035] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.887678 2643 consensus_queue.cc:1035] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.888888 2638 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: TxnStatusTablet state changed. Reason: New leader ffa91c44742f47bca2bb696f21777c7c. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.888972 2638 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.888897 2637 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: TxnStatusTablet state changed. Reason: New leader ffa91c44742f47bca2bb696f21777c7c. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.888995 2637 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.889892 2640 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.889981 2640 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:36.890019 2640 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.890051 2640 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P ffa91c44742f47bca2bb696f21777c7c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:36:36.890023 2637 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.890098 2637 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 1cc8d3e1dd2d47b1abb8194d10c3924f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.890304 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332981132847767552 to 7332981132803977216. Current Snapshot: MvccSnapshot[applied={T|T < 7332981132847767552 or (T in {7332981132847767552})}]
I20260924 19:36:36.891047 2660 mvcc.cc:205] Tried to move back new op lower bound from 7332981132847767552 to 7332981132803977216. Current Snapshot: MvccSnapshot[applied={T|T < 7332981132847767552}]
I20260924 19:36:36.891055 2659 mvcc.cc:205] Tried to move back new op lower bound from 7332981132847767552 to 7332981132803977216. Current Snapshot: MvccSnapshot[applied={T|T < 7332981132847767552}]
I20260924 19:36:36.891151 2638 tablet_replica.cc:439] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } } }
I20260924 19:36:36.891227 2638 tablet_replica.cc:442] T 23dab5a2ca5146eaaea7ed23385a3ece P 59fbfd39d6ff4cb4bdc919ab049b42bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:36:36.894604 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.892323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.0.98.2:32819)
0924 19:36:36.892389 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.892390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.892397 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:36.892414 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.892415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.892422 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.892464 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.892605 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.893032 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.894076 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.894245 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.894300 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.894439 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.894461 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.894488 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.894548 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.894561 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:36:36.898918 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.896847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55993 (local address 127.0.98.2:32819)
0924 19:36:36.896879 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.896881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.896988 (+ 107us) server_negotiation.cc:407] Connection header received
0924 19:36:36.897041 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.897041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.897050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.897087 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.897193 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.897611 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.898327 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.898493 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.898541 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.898684 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.898713 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.898752 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.898800 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.898832 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":15}
I20260924 19:36:36.898970 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.896807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.2:55993)
0924 19:36:36.896931 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.896941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.896992 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.897097 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.897098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.897099 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:36.897146 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.897147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.897670 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.897671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.898255 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.898257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.898543 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.898544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.898602 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.898616 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.898757 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.898758 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.898797 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.898855 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":13,"thread_start_us":73,"threads_started":1}
I20260924 19:36:36.900727 2286 raft_consensus.cc:1261] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.900743 2566 raft_consensus.cc:1261] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.900921 2670 consensus_queue.cc:1035] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.901047 2643 consensus_queue.cc:1035] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.914741 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:36.914871 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:36.915556 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Sending DeleteTablet for 3 replicas of tablet 8ec519d7fef34599a0f1a82d02239176
I20260924 19:36:36.915875 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec519d7fef34599a0f1a82d02239176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32972
I20260924 19:36:36.915871 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec519d7fef34599a0f1a82d02239176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260924 19:36:36.915956 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec519d7fef34599a0f1a82d02239176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260924 19:36:36.916148 2675 tablet_replica.cc:330] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: stopping tablet replica
I20260924 19:36:36.916265 2675 raft_consensus.cc:2229] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:36.916325 2675 raft_consensus.cc:2258] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:36.916311 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:36.916460 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:36.917009 2675 ts_tablet_manager.cc:1905] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:36.917496 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 46159e64aab844a082fe4834eaf7303c (DEFAULT_TABLE table=test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.917621 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46159e64aab844a082fe4834eaf7303c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.917651 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 46159e64aab844a082fe4834eaf7303c (DEFAULT_TABLE table=test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.917752 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46159e64aab844a082fe4834eaf7303c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.918922 2633 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Bootstrap starting.
I20260924 19:36:36.918944 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 46159e64aab844a082fe4834eaf7303c (DEFAULT_TABLE table=test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:36.919145 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46159e64aab844a082fe4834eaf7303c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:36.919148 2675 ts_tablet_manager.cc:1918] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:36.919322 2675 log.cc:1196] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/8ec519d7fef34599a0f1a82d02239176
I20260924 19:36:36.919569 2633 tablet_bootstrap.cc:654] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.919678 2675 ts_tablet_manager.cc:1939] T 8ec519d7fef34599a0f1a82d02239176 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting consensus metadata
I20260924 19:36:36.919909 2636 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Bootstrap starting.
I20260924 19:36:36.920318 2636 tablet_bootstrap.cc:654] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.920320 2632 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Bootstrap starting.
I20260924 19:36:36.920404 2113 catalog_manager.cc:4841] TS 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323): tablet 8ec519d7fef34599a0f1a82d02239176 (table test-table [id=0c43b28e3d214100ad43c679646617cf]) successfully deleted
I20260924 19:36:36.920657 2632 tablet_bootstrap.cc:654] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:36.920922 2636 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: No bootstrap required, opened a new log
I20260924 19:36:36.920970 2636 ts_tablet_manager.cc:1397] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:36.921113 2636 raft_consensus.cc:348] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.921197 2636 raft_consensus.cc:374] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.921216 2636 raft_consensus.cc:729] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.921243 2633 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: No bootstrap required, opened a new log
I20260924 19:36:36.921268 2636 consensus_queue.cc:260] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.921289 2633 ts_tablet_manager.cc:1397] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:36.921377 2636 ts_tablet_manager.cc:1428] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.921434 2633 raft_consensus.cc:348] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.921501 2633 raft_consensus.cc:374] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.921516 2633 raft_consensus.cc:729] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.921553 2633 consensus_queue.cc:260] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.921635 2633 ts_tablet_manager.cc:1428] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.924029 2676 tablet_replica.cc:330] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: stopping tablet replica
I20260924 19:36:36.924024 2677 tablet_replica.cc:330] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: stopping tablet replica
I20260924 19:36:36.924153 2677 raft_consensus.cc:2229] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:36.924153 2676 raft_consensus.cc:2229] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:36.924216 2677 raft_consensus.cc:2258] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:36.924266 2676 raft_consensus.cc:2258] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:36.924629 2632 tablet_bootstrap.cc:492] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: No bootstrap required, opened a new log
I20260924 19:36:36.924642 2676 ts_tablet_manager.cc:1905] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:36.924682 2632 ts_tablet_manager.cc:1397] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:36:36.924826 2632 raft_consensus.cc:348] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.924892 2632 raft_consensus.cc:374] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:36.924911 2632 raft_consensus.cc:729] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Initialized, Role: FOLLOWER
I20260924 19:36:36.924945 2632 consensus_queue.cc:260] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.925035 2632 ts_tablet_manager.cc:1428] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:36.926343 2676 ts_tablet_manager.cc:1918] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:36.926403 2676 log.cc:1196] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/8ec519d7fef34599a0f1a82d02239176
I20260924 19:36:36.926604 2676 ts_tablet_manager.cc:1939] T 8ec519d7fef34599a0f1a82d02239176 P ffa91c44742f47bca2bb696f21777c7c: Deleting consensus metadata
I20260924 19:36:36.926930 2115 catalog_manager.cc:4841] TS ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819): tablet 8ec519d7fef34599a0f1a82d02239176 (table test-table [id=0c43b28e3d214100ad43c679646617cf]) successfully deleted
I20260924 19:36:36.930001 2638 raft_consensus.cc:482] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:36.930079 2638 raft_consensus.cc:504] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.930389 2638 leader_election.cc:288] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:36.932303 2677 ts_tablet_manager.cc:1905] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:36.934021 2677 ts_tablet_manager.cc:1918] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:36.934095 2677 log.cc:1196] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/8ec519d7fef34599a0f1a82d02239176
I20260924 19:36:36.934157 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.930716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42929 (local address 127.0.98.3:35323)
0924 19:36:36.930752 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.930754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.930775 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:36:36.930870 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.930871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.930882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.930940 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.931085 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.931641 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.933214 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.933782 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.933852 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.933863 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.934049 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.934091 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:36.934365 2677 ts_tablet_manager.cc:1939] T 8ec519d7fef34599a0f1a82d02239176 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting consensus metadata
I20260924 19:36:36.934569 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.930537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35323 (local address 127.0.98.1:42929)
0924 19:36:36.930733 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.930742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.930781 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.930979 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.930980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.930981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.931030 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.931032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.931740 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.931741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.933129 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.933133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.933896 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.933897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.933972 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.933976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.934403 (+ 427us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.934489 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260924 19:36:36.934832 2113 catalog_manager.cc:4841] TS 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963): tablet 8ec519d7fef34599a0f1a82d02239176 (table test-table [id=0c43b28e3d214100ad43c679646617cf]) successfully deleted
I20260924 19:36:36.934964 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46159e64aab844a082fe4834eaf7303c" candidate_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" is_pre_election: true
I20260924 19:36:36.935040 2566 raft_consensus.cc:2454] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59fbfd39d6ff4cb4bdc919ab049b42bb in term 0.
I20260924 19:36:36.935159 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.930569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60599 (local address 127.0.98.2:32819)
0924 19:36:36.930626 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.930628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.930737 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:36:36.930837 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.930838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.930848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.930892 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:36.931047 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.931791 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.934276 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.934796 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.934841 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.934854 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.935057 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.935092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:36:36.935163 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.930534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.1:60599)
0924 19:36:36.930672 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.930683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.930766 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.930908 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.930909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.930911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:36.930982 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.930984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.931880 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.931881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.934159 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.934162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.934913 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.934914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.934986 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.934990 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.935043 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.935081 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1233,"thread_start_us":61,"threads_started":1}
I20260924 19:36:36.935267 2217 leader_election.cc:302] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc8d3e1dd2d47b1abb8194d10c3924f, 59fbfd39d6ff4cb4bdc919ab049b42bb; no voters:
I20260924 19:36:36.935379 2638 raft_consensus.cc:2781] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:36.935423 2638 raft_consensus.cc:482] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:36.935444 2638 raft_consensus.cc:3037] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.935388 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46159e64aab844a082fe4834eaf7303c" candidate_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa91c44742f47bca2bb696f21777c7c" is_pre_election: true
I20260924 19:36:36.935529 2426 raft_consensus.cc:2454] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59fbfd39d6ff4cb4bdc919ab049b42bb in term 0.
I20260924 19:36:36.936142 2638 raft_consensus.cc:504] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.936259 2638 leader_election.cc:288] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [CANDIDATE]: Term 1 election: Requested vote from peers ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:36.936416 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46159e64aab844a082fe4834eaf7303c" candidate_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa91c44742f47bca2bb696f21777c7c"
I20260924 19:36:36.936416 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46159e64aab844a082fe4834eaf7303c" candidate_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
I20260924 19:36:36.936491 2426 raft_consensus.cc:3037] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.936491 2566 raft_consensus.cc:3037] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:36.937142 2426 raft_consensus.cc:2454] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59fbfd39d6ff4cb4bdc919ab049b42bb in term 1.
I20260924 19:36:36.937152 2566 raft_consensus.cc:2454] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59fbfd39d6ff4cb4bdc919ab049b42bb in term 1.
I20260924 19:36:36.937302 2219 leader_election.cc:302] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fbfd39d6ff4cb4bdc919ab049b42bb, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:36.937415 2638 raft_consensus.cc:2781] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:36.937503 2638 raft_consensus.cc:686] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 LEADER]: Becoming Leader. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Running, Role: LEADER
I20260924 19:36:36.937563 2638 consensus_queue.cc:237] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:36.938206 2128 catalog_manager.cc:5495] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:36.946681 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.944347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.98.1:40963)
0924 19:36:36.944387 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.944389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.944398 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:36.944432 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.944432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.944450 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.944527 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.944603 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.945269 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.945965 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.946224 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.946291 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.946398 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.946516 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.946582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.946598 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.946619 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:36:36.951448 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.948919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47403 (local address 127.0.98.2:32819)
0924 19:36:36.948960 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.948962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.949063 (+ 101us) server_negotiation.cc:407] Connection header received
0924 19:36:36.949121 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.949122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.949132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.949179 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.949311 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.949774 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.950839 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.951075 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.951122 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.951280 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.951303 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.951334 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.951385 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.951400 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:36.951521 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.948874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.1:47403)
0924 19:36:36.949008 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.949017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.949068 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.949180 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.949181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.949182 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:36.949241 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.949243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.949858 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.949859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.950766 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.950769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.951135 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.951136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.951198 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.951212 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.951338 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.951339 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.951383 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.951424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260924 19:36:36.953354 2566 raft_consensus.cc:1261] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Refusing update from remote peer 59fbfd39d6ff4cb4bdc919ab049b42bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.953449 2426 raft_consensus.cc:1261] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Refusing update from remote peer 59fbfd39d6ff4cb4bdc919ab049b42bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:36.953686 2683 consensus_queue.cc:1035] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.953809 2638 consensus_queue.cc:1035] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:36.961086 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.958656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42691 (local address 127.0.98.1:40963)
0924 19:36:36.958700 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:36.958702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:36.958709 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:36.958727 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:36.958728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:36.958739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:36.958793 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:36.958894 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.959434 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.960466 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.960670 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.960723 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.960877 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.960908 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.960952 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:36.961005 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:36:36.961022 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:36:36.961095 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:36.958514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.98.2:42691)
0924 19:36:36.958550 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:36.958561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:36.958594 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:36.958798 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:36.958798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:36.958800 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:36.958836 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.958838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.959503 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.959504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.960395 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:36.960398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:36.960755 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:36.960756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:36.960810 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:36.960815 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:36.960958 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:36.960959 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:36.960995 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 19:36:36.961032 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 19:36:36.961561 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:36:38.481987 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:38.482091 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:38.482885 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Sending DeleteTablet for 3 replicas of tablet 46159e64aab844a082fe4834eaf7303c
I20260924 19:36:38.483130 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 46159e64aab844a082fe4834eaf7303c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260924 19:36:38.483186 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 46159e64aab844a082fe4834eaf7303c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32972
I20260924 19:36:38.483199 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 46159e64aab844a082fe4834eaf7303c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260924 19:36:38.484045 2716 tablet_replica.cc:330] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: stopping tablet replica
I20260924 19:36:38.484108 2716 raft_consensus.cc:2229] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:38.484138 2716 raft_consensus.cc:2258] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.484215 2714 tablet_replica.cc:330] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: stopping tablet replica
I20260924 19:36:38.484259 2714 raft_consensus.cc:2229] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:38.484336 2714 raft_consensus.cc:2258] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.484413 2715 tablet_replica.cc:330] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: stopping tablet replica
I20260924 19:36:38.484462 2715 raft_consensus.cc:2229] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:38.484493 2715 raft_consensus.cc:2258] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.484726 2715 ts_tablet_manager.cc:1905] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.485128 2716 ts_tablet_manager.cc:1905] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.486258 2715 ts_tablet_manager.cc:1918] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:36:38.486315 2715 log.cc:1196] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/46159e64aab844a082fe4834eaf7303c
I20260924 19:36:38.486394 2716 ts_tablet_manager.cc:1918] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:36:38.486449 2716 log.cc:1196] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/46159e64aab844a082fe4834eaf7303c
I20260924 19:36:38.486512 2715 ts_tablet_manager.cc:1939] T 46159e64aab844a082fe4834eaf7303c P ffa91c44742f47bca2bb696f21777c7c: Deleting consensus metadata
I20260924 19:36:38.486653 2716 ts_tablet_manager.cc:1939] T 46159e64aab844a082fe4834eaf7303c P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting consensus metadata
I20260924 19:36:38.486770 2115 catalog_manager.cc:4841] TS ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819): tablet 46159e64aab844a082fe4834eaf7303c (table test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]) successfully deleted
I20260924 19:36:38.486910 2113 catalog_manager.cc:4841] TS 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323): tablet 46159e64aab844a082fe4834eaf7303c (table test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]) successfully deleted
I20260924 19:36:38.488097 2714 ts_tablet_manager.cc:1905] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.489285 2714 ts_tablet_manager.cc:1918] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:36:38.489331 2714 log.cc:1196] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/46159e64aab844a082fe4834eaf7303c
I20260924 19:36:38.489531 2714 ts_tablet_manager.cc:1939] T 46159e64aab844a082fe4834eaf7303c P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting consensus metadata
I20260924 19:36:38.489928 2113 catalog_manager.cc:4841] TS 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963): tablet 46159e64aab844a082fe4834eaf7303c (table test-table [id=2dc57f2f9fc74cd18d1557c9e0fa1b0d]) successfully deleted
Sep 24 19:36:38 dist-test-slave-6kf3 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278598, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:36:38 dist-test-slave-6kf3 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278598, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:38.505028 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.498615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.0.98.62:45617)
0924 19:36:38.498732 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.498734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.498743 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:38.498785 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.498786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.498795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.498849 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:38.499345 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.499843 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.500560 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.500746 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.500791 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.502767 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:38.502768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:38.502768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:38.502774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:38.504078 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:38.504368 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:38.504369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:38.504370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:38.504420 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:38.504574 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:38.504575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:38.504575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:38.504675 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:38.504761 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.504852 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.504933 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260924 19:36:38.506271 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:38.506425 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:38.507551 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 (DEFAULT_TABLE table=test-table [id=bdc7b9a24f534ab49c038a96811ce06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.507591 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 (DEFAULT_TABLE table=test-table [id=bdc7b9a24f534ab49c038a96811ce06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.507668 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea499bfd874005bd0c30d644d8b5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.507665 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 (DEFAULT_TABLE table=test-table [id=bdc7b9a24f534ab49c038a96811ce06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.507699 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea499bfd874005bd0c30d644d8b5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.507759 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea499bfd874005bd0c30d644d8b5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.508772 2725 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Bootstrap starting.
I20260924 19:36:38.508994 2726 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Bootstrap starting.
I20260924 19:36:38.509131 2727 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Bootstrap starting.
I20260924 19:36:38.509279 2725 tablet_bootstrap.cc:654] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.509281 2726 tablet_bootstrap.cc:654] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.509493 2727 tablet_bootstrap.cc:654] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.509821 2726 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: No bootstrap required, opened a new log
I20260924 19:36:38.509864 2726 ts_tablet_manager.cc:1397] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:38.509969 2726 raft_consensus.cc:348] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510025 2727 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: No bootstrap required, opened a new log
I20260924 19:36:38.510036 2726 raft_consensus.cc:374] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.510056 2726 raft_consensus.cc:729] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.510056 2727 ts_tablet_manager.cc:1397] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:38.510108 2726 consensus_queue.cc:260] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510164 2727 raft_consensus.cc:348] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510200 2726 ts_tablet_manager.cc:1428] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.510231 2727 raft_consensus.cc:374] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.510254 2727 raft_consensus.cc:729] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.510286 2727 consensus_queue.cc:260] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510421 2727 ts_tablet_manager.cc:1428] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.510421 2725 tablet_bootstrap.cc:492] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: No bootstrap required, opened a new log
I20260924 19:36:38.510462 2725 ts_tablet_manager.cc:1397] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:38.510578 2725 raft_consensus.cc:348] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510766 2725 raft_consensus.cc:374] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.510802 2725 raft_consensus.cc:729] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.510843 2725 consensus_queue.cc:260] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.510931 2725 ts_tablet_manager.cc:1428] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.741536 2700 raft_consensus.cc:482] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:38.741634 2700 raft_consensus.cc:504] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.741760 2700 leader_election.cc:288] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:38.742002 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea499bfd874005bd0c30d644d8b5a4" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" is_pre_election: true
I20260924 19:36:38.742020 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea499bfd874005bd0c30d644d8b5a4" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" is_pre_election: true
I20260924 19:36:38.742100 2286 raft_consensus.cc:2454] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:38.742121 2566 raft_consensus.cc:2454] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 0.
I20260924 19:36:38.742272 2357 leader_election.cc:302] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fbfd39d6ff4cb4bdc919ab049b42bb, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:38.742373 2700 raft_consensus.cc:2781] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:38.742442 2700 raft_consensus.cc:482] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:38.742456 2700 raft_consensus.cc:3037] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.743120 2700 raft_consensus.cc:504] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.743224 2700 leader_election.cc:288] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Requested vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323)
I20260924 19:36:38.743429 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea499bfd874005bd0c30d644d8b5a4" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f"
I20260924 19:36:38.743424 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea499bfd874005bd0c30d644d8b5a4" candidate_uuid: "ffa91c44742f47bca2bb696f21777c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
I20260924 19:36:38.743511 2286 raft_consensus.cc:3037] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.743513 2566 raft_consensus.cc:3037] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.744122 2286 raft_consensus.cc:2454] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:38.744181 2566 raft_consensus.cc:2454] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa91c44742f47bca2bb696f21777c7c in term 1.
I20260924 19:36:38.744304 2357 leader_election.cc:302] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fbfd39d6ff4cb4bdc919ab049b42bb, ffa91c44742f47bca2bb696f21777c7c; no voters:
I20260924 19:36:38.744406 2700 raft_consensus.cc:2781] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:38.744467 2700 raft_consensus.cc:686] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 1 LEADER]: Becoming Leader. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Running, Role: LEADER
I20260924 19:36:38.744514 2700 consensus_queue.cc:237] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.745031 2128 catalog_manager.cc:5495] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c reported cstate change: term changed from 0 to 1, leader changed from <none> to ffa91c44742f47bca2bb696f21777c7c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ffa91c44742f47bca2bb696f21777c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:38.793617 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.791607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.98.2:32819)
0924 19:36:38.791733 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.791735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.791741 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:36:38.791758 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.791759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.791767 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.791819 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:38.791921 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.792429 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.793118 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.793297 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.793338 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.793429 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.793474 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.793503 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.793543 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.793558 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260924 19:36:38.795816 2566 raft_consensus.cc:1261] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:38.795846 2286 raft_consensus.cc:1261] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Refusing update from remote peer ffa91c44742f47bca2bb696f21777c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:38.796039 2695 consensus_queue.cc:1035] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:38.796162 2671 consensus_queue.cc:1035] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:38.805138 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:38.805223 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:38.805753 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Sending DeleteTablet for 3 replicas of tablet 73ea499bfd874005bd0c30d644d8b5a4
I20260924 19:36:38.805972 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260924 19:36:38.806021 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32972
I20260924 19:36:38.806008 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ea499bfd874005bd0c30d644d8b5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260924 19:36:38.806094 2714 tablet_replica.cc:330] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: stopping tablet replica
I20260924 19:36:38.806113 2716 tablet_replica.cc:330] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: stopping tablet replica
I20260924 19:36:38.806157 2714 raft_consensus.cc:2229] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:38.806164 2716 raft_consensus.cc:2229] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:38.806185 2714 raft_consensus.cc:2258] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.806202 2716 raft_consensus.cc:2258] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.806241 2715 tablet_replica.cc:330] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: stopping tablet replica
I20260924 19:36:38.806303 2715 raft_consensus.cc:2229] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:38.806357 2715 raft_consensus.cc:2258] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:38.806344 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:38.806476 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:38.806541 2716 ts_tablet_manager.cc:1905] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.806708 2715 ts_tablet_manager.cc:1905] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.807668 2266 tablet_service.cc:1467] Processing CreateTablet for tablet e874bb1514f346c5ad132d368709f43b (DEFAULT_TABLE table=test-table [id=343b018807e74a0ba7e5c400c55f7c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.807772 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874bb1514f346c5ad132d368709f43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.808092 2716 ts_tablet_manager.cc:1918] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:38.808152 2716 log.cc:1196] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/73ea499bfd874005bd0c30d644d8b5a4
I20260924 19:36:38.808169 2715 ts_tablet_manager.cc:1918] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:38.808221 2715 log.cc:1196] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/73ea499bfd874005bd0c30d644d8b5a4
I20260924 19:36:38.808383 2716 ts_tablet_manager.cc:1939] T 73ea499bfd874005bd0c30d644d8b5a4 P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting consensus metadata
I20260924 19:36:38.808410 2715 ts_tablet_manager.cc:1939] T 73ea499bfd874005bd0c30d644d8b5a4 P ffa91c44742f47bca2bb696f21777c7c: Deleting consensus metadata
I20260924 19:36:38.808683 2113 catalog_manager.cc:4841] TS 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323): tablet 73ea499bfd874005bd0c30d644d8b5a4 (table test-table [id=bdc7b9a24f534ab49c038a96811ce06b]) successfully deleted
I20260924 19:36:38.808745 2546 tablet_service.cc:1467] Processing CreateTablet for tablet e874bb1514f346c5ad132d368709f43b (DEFAULT_TABLE table=test-table [id=343b018807e74a0ba7e5c400c55f7c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.808826 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874bb1514f346c5ad132d368709f43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.809125 2726 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb: Bootstrap starting.
I20260924 19:36:38.809440 2726 tablet_bootstrap.cc:654] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.809754 2727 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Bootstrap starting.
I20260924 19:36:38.809875 2115 catalog_manager.cc:4841] TS ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819): tablet 73ea499bfd874005bd0c30d644d8b5a4 (table test-table [id=bdc7b9a24f534ab49c038a96811ce06b]) successfully deleted
I20260924 19:36:38.809948 2406 tablet_service.cc:1467] Processing CreateTablet for tablet e874bb1514f346c5ad132d368709f43b (DEFAULT_TABLE table=test-table [id=343b018807e74a0ba7e5c400c55f7c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:38.810042 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874bb1514f346c5ad132d368709f43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:38.810138 2714 ts_tablet_manager.cc:1905] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:38.810180 2727 tablet_bootstrap.cc:654] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.810575 2726 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb: No bootstrap required, opened a new log
I20260924 19:36:38.810624 2726 ts_tablet_manager.cc:1397] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:38.810729 2726 raft_consensus.cc:348] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.810784 2726 raft_consensus.cc:374] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.810801 2726 raft_consensus.cc:729] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fbfd39d6ff4cb4bdc919ab049b42bb, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.810842 2726 consensus_queue.cc:260] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.810927 2726 ts_tablet_manager.cc:1428] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.811116 2725 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Bootstrap starting.
I20260924 19:36:38.811269 2727 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: No bootstrap required, opened a new log
I20260924 19:36:38.811304 2727 ts_tablet_manager.cc:1397] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:38.811373 2714 ts_tablet_manager.cc:1918] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:36:38.811403 2725 tablet_bootstrap.cc:654] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:38.811419 2714 log.cc:1196] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/73ea499bfd874005bd0c30d644d8b5a4
I20260924 19:36:38.811425 2727 raft_consensus.cc:348] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.811501 2727 raft_consensus.cc:374] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.811519 2727 raft_consensus.cc:729] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.811570 2714 ts_tablet_manager.cc:1939] T 73ea499bfd874005bd0c30d644d8b5a4 P 59fbfd39d6ff4cb4bdc919ab049b42bb: Deleting consensus metadata
I20260924 19:36:38.811563 2727 consensus_queue.cc:260] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.811641 2727 ts_tablet_manager.cc:1428] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.811978 2113 catalog_manager.cc:4841] TS 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963): tablet 73ea499bfd874005bd0c30d644d8b5a4 (table test-table [id=bdc7b9a24f534ab49c038a96811ce06b]) successfully deleted
I20260924 19:36:38.812072 2725 tablet_bootstrap.cc:492] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: No bootstrap required, opened a new log
I20260924 19:36:38.812122 2725 ts_tablet_manager.cc:1397] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:36:38.812220 2725 raft_consensus.cc:348] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.812288 2725 raft_consensus.cc:374] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:38.812301 2725 raft_consensus.cc:729] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa91c44742f47bca2bb696f21777c7c, State: Initialized, Role: FOLLOWER
I20260924 19:36:38.812331 2725 consensus_queue.cc:260] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.812417 2725 ts_tablet_manager.cc:1428] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:38.851883 2731 raft_consensus.cc:482] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:38.852012 2731 raft_consensus.cc:504] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.852499 2731 leader_election.cc:288] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819)
I20260924 19:36:38.855353 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.852649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.98.3:51811)
0924 19:36:38.852776 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:38.852788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:38.852840 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:38.853025 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:38.853025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:38.853028 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:38.853091 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.853093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.853778 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.853779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.854595 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.854597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.855112 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.855113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.855180 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.855183 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:38.855225 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:36:38.855266 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 19:36:38.855343 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.852711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51811 (local address 127.0.98.1:40963)
0924 19:36:38.852843 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.852845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.852853 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:38.852905 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.852906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.852916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.852968 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:38.853154 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.853712 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.854681 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.855047 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.855106 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.855117 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.855236 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.855275 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260924 19:36:38.855480 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.852839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37535 (local address 127.0.98.2:32819)
0924 19:36:38.852879 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.852881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.852977 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:36:38.853057 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.853058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.853070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.853121 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:38.853281 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.853915 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.854795 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.855156 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.855220 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.855231 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.855381 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.855414 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:36:38.855491 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.852675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.3:37535)
0924 19:36:38.852911 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:38.852920 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:38.852988 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:38.853127 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:38.853128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:38.853130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:38.853209 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.853211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.854004 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.854005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.854702 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.854704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.855227 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.855228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.855310 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.855313 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:38.855358 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:36:38.855398 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":133,"threads_started":1}
I20260924 19:36:38.855749 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874bb1514f346c5ad132d368709f43b" candidate_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" is_pre_election: true
I20260924 19:36:38.855746 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874bb1514f346c5ad132d368709f43b" candidate_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa91c44742f47bca2bb696f21777c7c" is_pre_election: true
I20260924 19:36:38.855845 2286 raft_consensus.cc:2454] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc8d3e1dd2d47b1abb8194d10c3924f in term 0.
I20260924 19:36:38.855845 2426 raft_consensus.cc:2454] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc8d3e1dd2d47b1abb8194d10c3924f in term 0.
I20260924 19:36:38.856035 2497 leader_election.cc:302] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc8d3e1dd2d47b1abb8194d10c3924f, 59fbfd39d6ff4cb4bdc919ab049b42bb; no voters:
I20260924 19:36:38.856204 2731 raft_consensus.cc:2781] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:38.856232 2731 raft_consensus.cc:482] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:38.856251 2731 raft_consensus.cc:3037] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.856897 2731 raft_consensus.cc:504] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.857033 2731 leader_election.cc:288] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [CANDIDATE]: Term 1 election: Requested vote from peers 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963), ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819)
I20260924 19:36:38.857174 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874bb1514f346c5ad132d368709f43b" candidate_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa91c44742f47bca2bb696f21777c7c"
I20260924 19:36:38.857180 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874bb1514f346c5ad132d368709f43b" candidate_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb"
I20260924 19:36:38.857247 2426 raft_consensus.cc:3037] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.857249 2286 raft_consensus.cc:3037] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:38.857795 2286 raft_consensus.cc:2454] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc8d3e1dd2d47b1abb8194d10c3924f in term 1.
I20260924 19:36:38.857806 2426 raft_consensus.cc:2454] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc8d3e1dd2d47b1abb8194d10c3924f in term 1.
I20260924 19:36:38.857964 2497 leader_election.cc:302] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc8d3e1dd2d47b1abb8194d10c3924f, 59fbfd39d6ff4cb4bdc919ab049b42bb; no voters:
I20260924 19:36:38.858036 2731 raft_consensus.cc:2781] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:38.858196 2731 raft_consensus.cc:686] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 LEADER]: Becoming Leader. State: Replica: 1cc8d3e1dd2d47b1abb8194d10c3924f, State: Running, Role: LEADER
I20260924 19:36:38.858263 2731 consensus_queue.cc:237] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } }
I20260924 19:36:38.858871 2127 catalog_manager.cc:5495] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc8d3e1dd2d47b1abb8194d10c3924f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35323 } health_report { overall_health: HEALTHY } } }
I20260924 19:36:38.870803 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.868775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.0.98.3:35323)
0924 19:36:38.868898 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.868899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.868907 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:38.868946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.868946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.868962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.869031 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:38.869113 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.869586 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.870210 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.870404 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.870449 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.870540 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.870640 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.870688 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.870703 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.870717 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260924 19:36:38.875204 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.873046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42633 (local address 127.0.98.2:32819)
0924 19:36:38.873121 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.873124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.873169 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:36:38.873243 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.873243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.873254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.873299 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:38.873416 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.873861 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.874569 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.874750 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.874800 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.874973 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.875002 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.875040 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.875087 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.875128 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:36:38.875231 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.872956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.3:42633)
0924 19:36:38.873115 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:38.873127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:38.873184 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:38.873305 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:38.873305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:38.873307 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:38.873366 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.873367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.873925 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.873926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.874501 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.874502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.874820 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.874821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.874877 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.874889 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.875047 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.875048 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:38.875086 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 19:36:38.875141 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 19:36:38.877036 2426 raft_consensus.cc:1261] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Refusing update from remote peer 1cc8d3e1dd2d47b1abb8194d10c3924f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:38.877065 2286 raft_consensus.cc:1261] T e874bb1514f346c5ad132d368709f43b P 59fbfd39d6ff4cb4bdc919ab049b42bb [term 1 FOLLOWER]: Refusing update from remote peer 1cc8d3e1dd2d47b1abb8194d10c3924f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:38.877252 2731 consensus_queue.cc:1035] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa91c44742f47bca2bb696f21777c7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:38.877383 2744 consensus_queue.cc:1035] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fbfd39d6ff4cb4bdc919ab049b42bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:38.883713 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.881595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47207 (local address 127.0.98.3:35323)
0924 19:36:38.881645 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:38.881647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:38.881742 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:36:38.881796 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:38.881797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:38.881807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:38.881850 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:38.881932 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.882365 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.883051 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.883307 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.883370 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.883501 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.883534 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.883578 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:38.883627 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 19:36:38.883648 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:36:38.883750 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:38.881533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35323 (local address 127.0.98.2:47207)
0924 19:36:38.881687 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:38.881695 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:38.881750 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:38.881859 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:38.881859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:38.881861 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:36:38.881898 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.881899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.882415 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.882415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.882978 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:38.882980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:38.883369 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:38.883370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:38.883428 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:38.883439 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:36:38.883585 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:36:38.883586 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:38.883627 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:36:38.883669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
W20260924 19:36:38.883992 2697 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:36:40.400484 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:40.400589 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:40.401398 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 695d943e761b4a6f9107baac333cc442: Sending DeleteTablet for 3 replicas of tablet e874bb1514f346c5ad132d368709f43b
I20260924 19:36:40.401644 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet e874bb1514f346c5ad132d368709f43b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260924 19:36:40.401647 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet e874bb1514f346c5ad132d368709f43b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32972
I20260924 19:36:40.401845 2779 tablet_replica.cc:330] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: stopping tablet replica
I20260924 19:36:40.401909 2779 raft_consensus.cc:2229] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:40.401948 2779 raft_consensus.cc:2258] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:40.401937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2201
I20260924 19:36:40.402346 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet e874bb1514f346c5ad132d368709f43b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260924 19:36:40.402535 2781 tablet_replica.cc:330] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: stopping tablet replica
I20260924 19:36:40.402621 2781 raft_consensus.cc:2229] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:40.402685 2779 ts_tablet_manager.cc:1905] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:40.402721 2781 raft_consensus.cc:2258] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:40.403779 2781 ts_tablet_manager.cc:1905] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:40.404021 2779 ts_tablet_manager.cc:1918] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:36:40.404151 2779 log.cc:1196] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/e874bb1514f346c5ad132d368709f43b
I20260924 19:36:40.404364 2779 ts_tablet_manager.cc:1939] T e874bb1514f346c5ad132d368709f43b P ffa91c44742f47bca2bb696f21777c7c: Deleting consensus metadata
I20260924 19:36:40.404664 2115 catalog_manager.cc:4841] TS ffa91c44742f47bca2bb696f21777c7c (127.0.98.2:32819): tablet e874bb1514f346c5ad132d368709f43b (table test-table [id=343b018807e74a0ba7e5c400c55f7c3f]) successfully deleted
I20260924 19:36:40.405392 2781 ts_tablet_manager.cc:1918] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:36:40.405442 2781 log.cc:1196] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/e874bb1514f346c5ad132d368709f43b
I20260924 19:36:40.405632 2781 ts_tablet_manager.cc:1939] T e874bb1514f346c5ad132d368709f43b P 1cc8d3e1dd2d47b1abb8194d10c3924f: Deleting consensus metadata
I20260924 19:36:40.406018 2113 catalog_manager.cc:4841] TS 1cc8d3e1dd2d47b1abb8194d10c3924f (127.0.98.3:35323): tablet e874bb1514f346c5ad132d368709f43b (table test-table [id=343b018807e74a0ba7e5c400c55f7c3f]) successfully deleted
W20260924 19:36:40.408280 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40963 (error 108)
I20260924 19:36:40.408804 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:40.408599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.0.1:59808)
0924 19:36:40.408717 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:40.408764 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 19:36:40.408834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2341
W20260924 19:36:40.408932 2113 catalog_manager.cc:4584] TS 59fbfd39d6ff4cb4bdc919ab049b42bb (127.0.98.1:40963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e874bb1514f346c5ad132d368709f43b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40963: connect: Connection refused (error 111)
I20260924 19:36:40.414090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2481
I20260924 19:36:40.419111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2097
2026-09-24T19:36:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5624 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:40 dist-test-slave-6kf3 krb5kdc[2786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:40 dist-test-slave-6kf3 krb5kdc[2786](info): set up 2 sockets
Sep 24 19:36:40 dist-test-slave-6kf3 krb5kdc[2786](info): commencing operation
krb5kdc: starting...
W20260924 19:36:41.367584 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:41.479005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:41.558064 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:41.558285 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:41.558307 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:41.559471 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:41.559513 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:41.559526 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:41.559540 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:41.559562 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:41.561136 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:41.561372 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:41.561594 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:41.566329 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.566310 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.566299 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:41.566358 2799 server_base.cc:1034] running on GCE node
I20260924 19:36:41.566747 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:41.567021 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:41.568171 2799 hybrid_clock.cc:648] HybridClock initialized: now 1790278601568154 us; error 32 us; skew 500 ppm
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:41.570576 2799 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:41.570979 2799 webserver.cc:460] Webserver started at http://127.0.98.62:46075/ using document root <none> and password file <none>
I20260924 19:36:41.571156 2799 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:41.571197 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:41.571285 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:41.572373 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "c980f953d2124c848c5ebe546b8c22e8"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "e8bfc501833e906b4f26fac54d5baa69"
server_key_iv: "4450615a32a3ee5bc2e6cce40889f25e"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.572674 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "c980f953d2124c848c5ebe546b8c22e8"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "e8bfc501833e906b4f26fac54d5baa69"
server_key_iv: "4450615a32a3ee5bc2e6cce40889f25e"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.573927 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:41.574621 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.574795 2799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:41.574891 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "c980f953d2124c848c5ebe546b8c22e8"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "e8bfc501833e906b4f26fac54d5baa69"
server_key_iv: "4450615a32a3ee5bc2e6cce40889f25e"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.574957 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:41.590390 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:41.592341 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:41.592461 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:41.596073 2799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37895
I20260924 19:36:41.596129 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37895 every 8 connection(s)
I20260924 19:36:41.596589 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:41.597237 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:41.599479 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8: Bootstrap starting.
I20260924 19:36:41.600069 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:41.600333 2867 log.cc:824] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:41.601066 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8: No bootstrap required, opened a new log
I20260924 19:36:41.602603 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260924 19:36:41.602747 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:41.602769 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c980f953d2124c848c5ebe546b8c22e8, State: Initialized, Role: FOLLOWER
I20260924 19:36:41.602918 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260924 19:36:41.602980 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:41.603000 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:41.603036 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:41.603753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2799
I20260924 19:36:41.603828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:41.603920 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260924 19:36:41.604070 392 external_mini_cluster.cc:1428] Setting key c295ef2ba914ba41650cd0ef67718043
I20260924 19:36:41.604068 2867 leader_election.cc:302] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c980f953d2124c848c5ebe546b8c22e8; no voters:
I20260924 19:36:41.604223 2867 leader_election.cc:288] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:41.604267 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:41.604444 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [term 1 LEADER]: Becoming Leader. State: Replica: c980f953d2124c848c5ebe546b8c22e8, State: Running, Role: LEADER
I20260924 19:36:41.604579 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:41.604592 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260924 19:36:41.604996 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c980f953d2124c848c5ebe546b8c22e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } } }
I20260924 19:36:41.604993 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c980f953d2124c848c5ebe546b8c22e8. Latest consensus state: current_term: 1 leader_uuid: "c980f953d2124c848c5ebe546b8c22e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980f953d2124c848c5ebe546b8c22e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } } }
I20260924 19:36:41.605157 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:41.605172 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:41.605991 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:41.606490 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:41.608229 2876 catalog_manager.cc:1347] Generated new cluster ID: 8be9f342cfa34de5b392b19f82427adc
I20260924 19:36:41.608283 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:41.615507 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:41.604602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.98.62:37895)
0924 19:36:41.604764 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:41.604775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:41.604932 (+ 157us) server_negotiation.cc:407] Connection header received
0924 19:36:41.605135 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:41.605143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:41.605205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:41.605350 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:41.606295 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.607512 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.608557 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.609014 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.609118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:41.611485 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:41.611490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:41.611490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:41.611506 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:41.613815 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.614188 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.614192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.614193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.614296 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.614510 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.614511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.614512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.614779 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:41.614945 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:41.615063 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 19:36:41.615315 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 19:36:41.619825 2876 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:41.620556 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:41.630955 2876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c980f953d2124c848c5ebe546b8c22e8: Generated new TSK 0
I20260924 19:36:41.631162 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:41.659013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:41.756946 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:41.757161 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:41.757189 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:41.758384 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:41.758430 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:41.758457 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:41.760028 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:41.760264 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:41.760494 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:41.765193 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.765219 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.765226 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:41.765504 2893 server_base.cc:1034] running on GCE node
I20260924 19:36:41.765704 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:41.765930 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:41.767063 2893 hybrid_clock.cc:648] HybridClock initialized: now 1790278601767055 us; error 31 us; skew 500 ppm
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:41.769560 2893 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:41.769995 2893 webserver.cc:460] Webserver started at http://127.0.98.1:33579/ using document root <none> and password file <none>
I20260924 19:36:41.770207 2893 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:41.770255 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:41.770375 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:41.771381 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "1dfdb78851a7475d8694b4ca06113dbb"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "0b195f6b86b8713f9e3348a7cbc344eb"
server_key_iv: "d715d4a5be155950913ffbba9b0375e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.771706 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1dfdb78851a7475d8694b4ca06113dbb"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "0b195f6b86b8713f9e3348a7cbc344eb"
server_key_iv: "d715d4a5be155950913ffbba9b0375e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.772992 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:41.773712 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.773870 2893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:41.773949 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "1dfdb78851a7475d8694b4ca06113dbb"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "0b195f6b86b8713f9e3348a7cbc344eb"
server_key_iv: "d715d4a5be155950913ffbba9b0375e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.774015 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:41.789093 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:41.790927 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:41.791064 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:41.791278 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:41.791564 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:41.791595 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.791622 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:41.791646 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.797129 2893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38005
I20260924 19:36:41.797186 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38005 every 8 connection(s)
I20260924 19:36:41.797482 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:41.804663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 2893
I20260924 19:36:41.804764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:41.805037 392 external_mini_cluster.cc:1428] Setting key 21337541ac925b15b419628de1e96ec1
I20260924 19:36:41.806875 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:41.798170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48215 (local address 127.0.98.62:37895)
0924 19:36:41.798223 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:41.798225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:41.798429 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:36:41.799343 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:41.799344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:41.799356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:41.799423 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:41.800048 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.800586 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.801643 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.801854 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.801917 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:41.803965 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:41.803966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:41.803966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:41.803974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:41.805504 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.805897 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.805898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.805899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.805973 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.806199 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.806200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.806200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.806343 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:41.806404 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:41.806671 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:36:41.806751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:36:41.807035 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:41.798149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.1:48215)
0924 19:36:41.798308 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:41.798343 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:41.799228 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:41.799482 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:41.799486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:41.799730 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:41.799916 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:41.799920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.800690 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.800691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:41.801551 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:41.801554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.801935 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.801936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:41.802049 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:41.802529 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:41.802537 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:41.803843 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:41.805570 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:41.805576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:41.805578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:41.805830 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:41.806047 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:41.806048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:41.806048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:41.806144 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:41.806404 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:41.806408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:41.806555 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:36:41.806638 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:36:41.806759 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260924 19:36:41.807559 3022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260924 19:36:41.807652 3022 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:41.807821 3022 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260924 19:36:41.808434 2831 ts_manager.cc:194] Registered new tserver with Master: 1dfdb78851a7475d8694b4ca06113dbb (127.0.98.1:38005)
I20260924 19:36:41.809144 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48215
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:41.828107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:41.905790 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:41.906009 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:41.906032 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:41.907199 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:41.907248 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:41.907272 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:41.908788 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:41.909021 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:41.909229 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:41.914315 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.914374 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:41.914383 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:41.914536 3027 server_base.cc:1034] running on GCE node
I20260924 19:36:41.914741 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:41.914973 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:41.916103 3027 hybrid_clock.cc:648] HybridClock initialized: now 1790278601916086 us; error 35 us; skew 500 ppm
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:41.923218 3027 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:41.923696 3027 webserver.cc:460] Webserver started at http://127.0.98.2:39381/ using document root <none> and password file <none>
I20260924 19:36:41.923902 3027 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:41.923974 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:41.924105 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:41.925127 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "e061db1508c84fc8bc08170eac84680b"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "5ae773835c0b1d1aa794f3825a61a078"
server_key_iv: "ce994efc4724b30818e5ceccc3e39f6a"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.925477 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e061db1508c84fc8bc08170eac84680b"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "5ae773835c0b1d1aa794f3825a61a078"
server_key_iv: "ce994efc4724b30818e5ceccc3e39f6a"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.926677 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:36:41.927409 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.927561 3027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:41.927652 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "e061db1508c84fc8bc08170eac84680b"
format_stamp: "Formatted at 2026-09-24 19:36:41 on dist-test-slave-6kf3"
server_key: "5ae773835c0b1d1aa794f3825a61a078"
server_key_iv: "ce994efc4724b30818e5ceccc3e39f6a"
server_key_version: "encryptionkey@0"
I20260924 19:36:41.927713 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:41.947494 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:41.949674 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:41.949812 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:41.950021 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:41.950316 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:41.950352 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.950381 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:41.950403 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:41.955746 3027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41805
I20260924 19:36:41.955842 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41805 every 8 connection(s)
I20260924 19:36:41.956118 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:41 dist-test-slave-6kf3 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:41.962718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3027
I20260924 19:36:41.962795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:41.963006 392 external_mini_cluster.cc:1428] Setting key 70cd59a9762137308dbed9a8704b8a52
I20260924 19:36:41.965440 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:41.956789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37323 (local address 127.0.98.62:37895)
0924 19:36:41.956838 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:41.956840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:41.957038 (+ 198us) server_negotiation.cc:407] Connection header received
0924 19:36:41.957918 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:41.957919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:41.957930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:41.957987 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:41.958563 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.959057 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.960091 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.960293 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.960361 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:41.962656 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:41.962658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:41.962659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:41.962666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:41.964145 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.964540 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.964541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.964542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.964603 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:41.964825 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:41.964826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:41.964827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:41.964932 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:41.964987 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:41.965260 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:36:41.965343 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:36:41.965639 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:41.956764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.2:37323)
0924 19:36:41.956913 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:41.956962 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:41.957832 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:41.958052 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:41.958055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:41.958303 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:41.958497 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:41.958501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.959123 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.959124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:41.959998 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:41.960001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:41.960366 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:41.960367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:41.960479 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:41.961025 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:41.961034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:41.962536 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:41.964226 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:41.964235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:41.964238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:41.964470 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:41.964655 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:41.964656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:41.964656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:41.964766 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:41.964984 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:41.964991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:41.965139 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:36:41.965229 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:36:41.965407 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260924 19:36:41.966252 3156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260924 19:36:41.966367 3156 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:41.966547 3156 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260924 19:36:41.966912 2831 ts_manager.cc:194] Registered new tserver with Master: e061db1508c84fc8bc08170eac84680b (127.0.98.2:41805)
I20260924 19:36:41.967260 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37323
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:41.986194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:42.065337 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:42.065553 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:42.065584 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:42.066730 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:42.066777 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:42.066812 3161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:42.068356 3161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:42.068603 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:42.068812 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:42.073558 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:42.073577 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:42.073578 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:42.073784 3161 server_base.cc:1034] running on GCE node
I20260924 19:36:42.073980 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:42.074191 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:42.075328 3161 hybrid_clock.cc:648] HybridClock initialized: now 1790278602075316 us; error 30 us; skew 500 ppm
Sep 24 19:36:42 dist-test-slave-6kf3 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:42.077924 3161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:42.078343 3161 webserver.cc:460] Webserver started at http://127.0.98.3:41721/ using document root <none> and password file <none>
I20260924 19:36:42.078572 3161 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:42.078622 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:42.078743 3161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:42.079773 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "7dadc265aa014d14a1ce28de44667fc0"
format_stamp: "Formatted at 2026-09-24 19:36:42 on dist-test-slave-6kf3"
server_key: "6673dcd3310a9af206cf2a2f3fedb934"
server_key_iv: "0f2952523465a183a7edae587b1ce2ac"
server_key_version: "encryptionkey@0"
I20260924 19:36:42.080169 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7dadc265aa014d14a1ce28de44667fc0"
format_stamp: "Formatted at 2026-09-24 19:36:42 on dist-test-slave-6kf3"
server_key: "6673dcd3310a9af206cf2a2f3fedb934"
server_key_iv: "0f2952523465a183a7edae587b1ce2ac"
server_key_version: "encryptionkey@0"
I20260924 19:36:42.081449 3161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:42.082111 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:42.082247 3161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:42.082327 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "7dadc265aa014d14a1ce28de44667fc0"
format_stamp: "Formatted at 2026-09-24 19:36:42 on dist-test-slave-6kf3"
server_key: "6673dcd3310a9af206cf2a2f3fedb934"
server_key_iv: "0f2952523465a183a7edae587b1ce2ac"
server_key_version: "encryptionkey@0"
I20260924 19:36:42.082391 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:42.102814 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:42.105065 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:42.105204 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:42.105420 3161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:42.105742 3161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:42.105777 3161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:42.105805 3161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:42.105829 3161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:42.111374 3161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45109
I20260924 19:36:42.111433 3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45109 every 8 connection(s)
I20260924 19:36:42.111730 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:36:42 dist-test-slave-6kf3 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:42.120486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3161
I20260924 19:36:42.120587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:42.120751 392 external_mini_cluster.cc:1428] Setting key 4c59f6f91b20b0d82ce5000515c7931e
I20260924 19:36:42.120949 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:42.112449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44969 (local address 127.0.98.62:37895)
0924 19:36:42.112505 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:42.112507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:42.112692 (+ 185us) server_negotiation.cc:407] Connection header received
0924 19:36:42.113645 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:42.113647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:42.113658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:42.113718 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:42.114356 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:42.114840 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:42.115972 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:42.116180 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:42.116231 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:42.118348 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:42.118349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:42.118349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:42.118356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:42.119578 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:42.120048 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:42.120049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:42.120049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:42.120096 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:42.120292 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:42.120292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:42.120293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:42.120402 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:42.120467 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:42.120772 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:36:42.120854 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:36:42.121132 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:42.112417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.3:44969)
0924 19:36:42.112572 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:42.112614 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:42.113519 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:42.113788 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:42.113792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:42.114086 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:42.114279 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:42.114283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:42.114917 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:42.114917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:42.115881 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:42.115884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:42.116250 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:42.116251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:42.116366 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:42.116868 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:42.116877 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:42.118267 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:42.119662 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:42.119668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:42.119670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:42.119988 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:42.120146 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:42.120147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:42.120147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:42.120246 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:42.120462 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:42.120470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:42.120642 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 19:36:42.120743 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:36:42.120901 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 19:36:42.121654 3290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260924 19:36:42.121745 3290 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:42.121912 3290 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260924 19:36:42.122279 2831 ts_manager.cc:194] Registered new tserver with Master: 7dadc265aa014d14a1ce28de44667fc0 (127.0.98.3:45109)
I20260924 19:36:42.122644 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44969
I20260924 19:36:42.132442 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:36:42.142726 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:42.141876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.62:37895)
0924 19:36:42.141948 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:42.141950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:42.141995 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:36:42.142048 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:42.142048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:42.142057 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:42.142114 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:42.142648 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45042: BlockingRecv error: recv got EOF from 127.0.0.1:45042 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:36:42.143522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2893
I20260924 19:36:42.147294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3027
I20260924 19:36:42.151111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3161
I20260924 19:36:42.154548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 2799
2026-09-24T19:36:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1734 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:42 dist-test-slave-6kf3 krb5kdc[3302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:42 dist-test-slave-6kf3 krb5kdc[3302](info): set up 2 sockets
Sep 24 19:36:42 dist-test-slave-6kf3 krb5kdc[3302](info): commencing operation
krb5kdc: starting...
W20260924 19:36:43.099531 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:43.208273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:43.287137 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:43.287345 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:43.287375 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:43.288581 3315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:43.288625 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:43.288643 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:43.288661 3315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:43.288684 3315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:43.290207 3315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:43.290445 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:43.290647 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:43.295238 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:43.295219 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:43.295346 3315 server_base.cc:1034] running on GCE node
W20260924 19:36:43.295200 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:43.295632 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:43.295917 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:43.297083 3315 hybrid_clock.cc:648] HybridClock initialized: now 1790278603297064 us; error 34 us; skew 500 ppm
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:43.299423 3315 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:43.299849 3315 webserver.cc:460] Webserver started at http://127.0.98.62:38399/ using document root <none> and password file <none>
I20260924 19:36:43.300087 3315 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:43.300148 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:43.300274 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:43.301331 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "380ef09b6716456cb95027d5fd119dc7"
format_stamp: "Formatted at 2026-09-24 19:36:43 on dist-test-slave-6kf3"
server_key: "062419787e6faae060b21c5e93b8a28a"
server_key_iv: "c51296b69f3b8f44746a8f55b76d6871"
server_key_version: "encryptionkey@0"
I20260924 19:36:43.301654 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "380ef09b6716456cb95027d5fd119dc7"
format_stamp: "Formatted at 2026-09-24 19:36:43 on dist-test-slave-6kf3"
server_key: "062419787e6faae060b21c5e93b8a28a"
server_key_iv: "c51296b69f3b8f44746a8f55b76d6871"
server_key_version: "encryptionkey@0"
I20260924 19:36:43.302898 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:36:43.303556 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:43.303702 3315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:43.303784 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "380ef09b6716456cb95027d5fd119dc7"
format_stamp: "Formatted at 2026-09-24 19:36:43 on dist-test-slave-6kf3"
server_key: "062419787e6faae060b21c5e93b8a28a"
server_key_iv: "c51296b69f3b8f44746a8f55b76d6871"
server_key_version: "encryptionkey@0"
I20260924 19:36:43.303854 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:43.345322 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:43.347234 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:43.347386 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:43.351150 3315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37319
I20260924 19:36:43.351181 3382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37319 every 8 connection(s)
I20260924 19:36:43.351512 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:43.352406 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:43.352771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3315
I20260924 19:36:43.352861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:43.353034 392 external_mini_cluster.cc:1428] Setting key 2c0e3352544580ca4a983674b99288a0
I20260924 19:36:43.355304 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7: Bootstrap starting.
I20260924 19:36:43.355877 3383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:43.356149 3383 log.cc:824] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:43.356747 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7: No bootstrap required, opened a new log
I20260924 19:36:43.358367 3383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 19:36:43.358505 3383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:43.358536 3383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380ef09b6716456cb95027d5fd119dc7, State: Initialized, Role: FOLLOWER
I20260924 19:36:43.358672 3383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 19:36:43.358748 3383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:43.358789 3383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:43.358841 3383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:43.359417 3383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 19:36:43.359512 3383 leader_election.cc:302] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380ef09b6716456cb95027d5fd119dc7; no voters:
I20260924 19:36:43.359685 3383 leader_election.cc:288] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:43.359723 3388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:43.359866 3388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 380ef09b6716456cb95027d5fd119dc7, State: Running, Role: LEADER
I20260924 19:36:43.359972 3383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:43.360001 3388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 19:36:43.360302 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380ef09b6716456cb95027d5fd119dc7. Latest consensus state: current_term: 1 leader_uuid: "380ef09b6716456cb95027d5fd119dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } } }
I20260924 19:36:43.360375 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:43.360584 3394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:43.361104 3394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:43.360531 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380ef09b6716456cb95027d5fd119dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380ef09b6716456cb95027d5fd119dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } } }
I20260924 19:36:43.361661 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:43.362962 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:43.353519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.0.98.62:37319)
0924 19:36:43.353667 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:43.353675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:43.353701 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:36:43.354132 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:43.354139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:43.354189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:43.354324 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:43.355058 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:43.355992 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:43.356730 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:43.357191 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:43.357295 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:43.359559 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:43.359563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:43.359564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:43.359577 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:43.361443 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:43.361809 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:43.361813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:43.361813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:43.361927 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:43.362121 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:43.362122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:43.362122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:43.362349 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:43.362484 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:43.362641 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:36:43.362778 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 19:36:43.363210 3394 catalog_manager.cc:1347] Generated new cluster ID: 8cdaf98e919c44e88dff271087e0ca5a
I20260924 19:36:43.363263 3394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:43.363920 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:36:43.388633 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:43.384535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.0.98.62:37319)
0924 19:36:43.384588 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:43.384590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:43.384700 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:36:43.384757 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:43.384758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:43.384770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:43.384837 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:43.385229 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:43.386008 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:43.387015 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:43.387358 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:43.387421 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:43.388132 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:43.388134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:43.388134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:43.388372 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:43.388392 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:43.388399 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:43.388453 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:43.388511 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 19:36:43.388537 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:36:43.390741 3394 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:43.391245 3394 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:43.396605 3394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380ef09b6716456cb95027d5fd119dc7: Generated new TSK 0
I20260924 19:36:43.396771 3394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:36:43.403839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3315
2026-09-24T19:36:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1248 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3416](info): set up 2 sockets
Sep 24 19:36:43 dist-test-slave-6kf3 krb5kdc[3416](info): commencing operation
krb5kdc: starting...
W20260924 19:36:44.347501 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:44.456261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:44.534760 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:44.534976 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:44.535005 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:44.536202 3429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:44.536243 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:44.536262 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:44.536278 3429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:44.536293 3429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:44.537770 3429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:44.538013 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:44.538228 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:44.543061 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:44.542973 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:44.543174 3429 server_base.cc:1034] running on GCE node
W20260924 19:36:44.542971 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:44.543471 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:44.543748 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:44.544883 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790278604544862 us; error 47 us; skew 500 ppm
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:44.547235 3429 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:44.547683 3429 webserver.cc:460] Webserver started at http://127.0.98.62:43289/ using document root <none> and password file <none>
I20260924 19:36:44.547895 3429 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:44.547966 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:44.548090 3429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:44.549115 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "0b06475f37d646b3bdf016359684de05"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "f760fc328adfbaa9ac0cf6672d99276a"
server_key_iv: "a94104f0b0a84864cd2f8f63564eb7e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.549427 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b06475f37d646b3bdf016359684de05"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "f760fc328adfbaa9ac0cf6672d99276a"
server_key_iv: "a94104f0b0a84864cd2f8f63564eb7e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.550746 3429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:44.551406 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.551553 3429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:44.551635 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "0b06475f37d646b3bdf016359684de05"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "f760fc328adfbaa9ac0cf6672d99276a"
server_key_iv: "a94104f0b0a84864cd2f8f63564eb7e4"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.551694 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:44.569797 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:44.571542 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:44.571676 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:44.575366 3429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36943
I20260924 19:36:44.575404 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36943 every 8 connection(s)
I20260924 19:36:44.575732 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:44.576723 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:44.578892 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: Bootstrap starting.
I20260924 19:36:44.579421 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:44.579672 3497 log.cc:824] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:44.580317 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: No bootstrap required, opened a new log
I20260924 19:36:44.580565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3429
I20260924 19:36:44.580641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:44.580799 392 external_mini_cluster.cc:1428] Setting key dd4ad618a0f590838626dc4d07b30d40
I20260924 19:36:44.582326 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } }
I20260924 19:36:44.582494 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:44.582530 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b06475f37d646b3bdf016359684de05, State: Initialized, Role: FOLLOWER
I20260924 19:36:44.582707 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } }
I20260924 19:36:44.582788 3497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:44.582811 3497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:44.582856 3497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:44.583871 3497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } }
I20260924 19:36:44.583987 3497 leader_election.cc:302] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b06475f37d646b3bdf016359684de05; no voters:
I20260924 19:36:44.584132 3497 leader_election.cc:288] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:44.584169 3502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:44.584306 3502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [term 1 LEADER]: Becoming Leader. State: Replica: 0b06475f37d646b3bdf016359684de05, State: Running, Role: LEADER
I20260924 19:36:44.584355 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:44.584445 3502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } }
I20260924 19:36:44.585788 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b06475f37d646b3bdf016359684de05. Latest consensus state: current_term: 1 leader_uuid: "0b06475f37d646b3bdf016359684de05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } } }
I20260924 19:36:44.585858 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b06475f37d646b3bdf016359684de05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b06475f37d646b3bdf016359684de05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36943 } } }
I20260924 19:36:44.585923 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [sys.catalog]: This master's current role is: LEADER
W20260924 19:36:44.586045 3517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:44.586102 3517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:44.585980 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:44.586212 3519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:44.586338 3519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:44.587728 3519 catalog_manager.cc:1347] Generated new cluster ID: c4e47de5ee4d4c2ab8fcc9880549015f
I20260924 19:36:44.587778 3519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:44.591609 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:44.581267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.0.98.62:36943)
0924 19:36:44.581448 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:44.581458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:44.581496 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:36:44.582291 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:44.582300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:44.582366 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:44.582516 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:44.583148 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.584551 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.585532 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.586143 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.586252 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:44.588955 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:44.588958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:44.588958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:44.588968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:44.590354 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.590687 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.590689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.590689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.590765 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.590906 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.590906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.590907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.591123 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:44.591212 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:44.591315 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:36:44.591413 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260924 19:36:44.605844 3519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:44.606385 3519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:44.615792 3519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b06475f37d646b3bdf016359684de05: Generated new TSK 0
I20260924 19:36:44.616008 3519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:44.642211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:44.725136 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:44.725359 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:44.725384 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:44.726594 3523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:44.726644 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:44.726671 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:44.728164 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:44.728404 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:44.728634 3523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:44.733561 3531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:44.733557 3528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:44.733603 3529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:44.734377 3523 server_base.cc:1034] running on GCE node
I20260924 19:36:44.734586 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:44.734812 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:44.735967 3523 hybrid_clock.cc:648] HybridClock initialized: now 1790278604735952 us; error 31 us; skew 500 ppm
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:44.738371 3523 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:44.738811 3523 webserver.cc:460] Webserver started at http://127.0.98.1:33549/ using document root <none> and password file <none>
I20260924 19:36:44.739007 3523 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:44.739064 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:44.739190 3523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:44.740300 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f26e31b10fb4acca6e4c89629e2e6dd"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "475300c67333acdfa33329404c198edd"
server_key_iv: "7a5cc28d4b6bd64b99beef6eade0d8b2"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.740628 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f26e31b10fb4acca6e4c89629e2e6dd"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "475300c67333acdfa33329404c198edd"
server_key_iv: "7a5cc28d4b6bd64b99beef6eade0d8b2"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.741811 3523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:44.742460 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.742602 3523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:44.742681 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "4f26e31b10fb4acca6e4c89629e2e6dd"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "475300c67333acdfa33329404c198edd"
server_key_iv: "7a5cc28d4b6bd64b99beef6eade0d8b2"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.742738 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:44.757563 3523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:44.759477 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:44.759609 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:44.759838 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:44.760197 3523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:44.760232 3523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.760262 3523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:44.760286 3523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.765666 3523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35867
I20260924 19:36:44.765733 3651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35867 every 8 connection(s)
I20260924 19:36:44.766012 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:44.766729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3523
I20260924 19:36:44.766816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:44.767020 392 external_mini_cluster.cc:1428] Setting key 6d792aec591986f58919036a6633a4f7
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:36:44.775455 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:44.766706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57757 (local address 127.0.98.62:36943)
0924 19:36:44.766761 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:44.766763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:44.767019 (+ 256us) server_negotiation.cc:407] Connection header received
0924 19:36:44.767613 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:44.767615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:44.767628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:44.767683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:44.768876 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.769370 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.770346 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.770550 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.770609 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:44.772872 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:44.772873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:44.772873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:44.772883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:44.774195 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.774642 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.774643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.774643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.774693 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.774898 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.774898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.774898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.774994 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:44.775038 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:44.775312 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:36:44.775387 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:36:44.775619 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:44.766669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36943 (local address 127.0.98.1:57757)
0924 19:36:44.766873 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:44.766923 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:44.767508 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:44.767763 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:44.767767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:44.768511 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:44.768795 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:44.768801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.769456 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.769457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:44.770265 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:44.770267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.770623 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.770624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:44.770768 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:44.771248 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:44.771257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:44.772782 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:44.774274 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:44.774281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:44.774283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:44.774576 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:44.774755 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:44.774755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:44.774756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:44.774855 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:44.775041 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:44.775046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:44.775200 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 19:36:44.775284 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:36:44.775407 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260924 19:36:44.776095 3652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36943
I20260924 19:36:44.776181 3652 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 19:36:44.776384 3652 heartbeater.cc:511] Master 127.0.98.62:36943 requested a full tablet report, sending...
I20260924 19:36:44.776934 3461 ts_manager.cc:194] Registered new tserver with Master: 4f26e31b10fb4acca6e4c89629e2e6dd (127.0.98.1:35867)
I20260924 19:36:44.777768 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:44.789109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:44.875768 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:44.876010 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:44.876034 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:44.877161 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:44.877202 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:44.877225 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:44.878611 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:44.878793 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:44.878979 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:44.883679 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:44.883669 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:44.883669 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:44.883904 3657 server_base.cc:1034] running on GCE node
I20260924 19:36:44.884116 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:44.884370 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:44.885500 3657 hybrid_clock.cc:648] HybridClock initialized: now 1790278604885479 us; error 33 us; skew 500 ppm
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:44.887859 3657 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:44.888355 3657 webserver.cc:460] Webserver started at http://127.0.98.2:33069/ using document root <none> and password file <none>
I20260924 19:36:44.888586 3657 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:44.888631 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:44.888756 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:44.889820 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "c7f66200703e4361b7061e6657642317"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "64c3061c85da1990bc58ff9aea5dd3c9"
server_key_iv: "91e684428036e813753823325b380413"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.890141 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c7f66200703e4361b7061e6657642317"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "64c3061c85da1990bc58ff9aea5dd3c9"
server_key_iv: "91e684428036e813753823325b380413"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.891345 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:44.892061 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.892198 3657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:44.892280 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "c7f66200703e4361b7061e6657642317"
format_stamp: "Formatted at 2026-09-24 19:36:44 on dist-test-slave-6kf3"
server_key: "64c3061c85da1990bc58ff9aea5dd3c9"
server_key_iv: "91e684428036e813753823325b380413"
server_key_version: "encryptionkey@0"
I20260924 19:36:44.892334 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:44.908310 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:44.910086 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:44.910200 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:44.910388 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:44.910650 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:44.910681 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.910701 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:44.910717 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:44.916175 3657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37675
I20260924 19:36:44.916247 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37675 every 8 connection(s)
I20260924 19:36:44.916535 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:44 dist-test-slave-6kf3 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:44.924896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3657
I20260924 19:36:44.924990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:44.925130 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:44.917208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54633 (local address 127.0.98.62:36943)
0924 19:36:44.917260 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:44.917263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:44.917444 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:36:44.917955 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:44.917956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:44.917968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:44.918020 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:44.918966 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.919463 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.920516 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.920702 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.920757 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:44.922779 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:44.922781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:44.922782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:44.922790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:44.923854 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.924249 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.924250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.924250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.924302 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:44.924510 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:44.924511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:44.924511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:44.924622 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:44.924678 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:44.924936 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:36:44.925024 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:36:44.925185 392 external_mini_cluster.cc:1428] Setting key 4ee92c36aff033ba9672d5b0c077f9e3
I20260924 19:36:44.925295 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:44.917186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36943 (local address 127.0.98.2:54633)
0924 19:36:44.917330 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:44.917366 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:44.917857 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:44.918093 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:44.918097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:44.918665 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:44.918864 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:44.918868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.919554 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.919555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:44.920429 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:44.920431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:44.920769 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:44.920770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:44.920885 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:44.921326 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:44.921334 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:44.922688 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:44.923926 (+ 1238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:44.923960 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:44.923963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:44.924182 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:44.924353 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:44.924354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:44.924354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:44.924457 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:44.924674 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:44.924680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:44.924822 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 19:36:44.924912 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:36:44.925073 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260924 19:36:44.925760 3786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36943
I20260924 19:36:44.925853 3786 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:44.926028 3786 heartbeater.cc:511] Master 127.0.98.62:36943 requested a full tablet report, sending...
I20260924 19:36:44.926385 3461 ts_manager.cc:194] Registered new tserver with Master: c7f66200703e4361b7061e6657642317 (127.0.98.2:37675)
I20260924 19:36:44.926749 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54633
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:44.947163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:45.025749 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:45.025970 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:45.026003 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:45.027169 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:45.027210 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:45.027244 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:45.028695 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:45.028931 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:45.029145 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:45.033797 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:45.033790 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:45.033806 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:45.033994 3791 server_base.cc:1034] running on GCE node
I20260924 19:36:45.034190 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:45.034385 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:45.035513 3791 hybrid_clock.cc:648] HybridClock initialized: now 1790278605035499 us; error 28 us; skew 500 ppm
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:45.037783 3791 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:45.038201 3791 webserver.cc:460] Webserver started at http://127.0.98.3:38103/ using document root <none> and password file <none>
I20260924 19:36:45.038403 3791 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:45.038443 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:45.038566 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:45.039579 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "b8c0c894ab844c3fb6047a8c7e089928"
format_stamp: "Formatted at 2026-09-24 19:36:45 on dist-test-slave-6kf3"
server_key: "f1695fd5b5b905f0d152255f693a8294"
server_key_iv: "8c0eb09308bd9af8fd159b2144c36c5c"
server_key_version: "encryptionkey@0"
I20260924 19:36:45.039893 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b8c0c894ab844c3fb6047a8c7e089928"
format_stamp: "Formatted at 2026-09-24 19:36:45 on dist-test-slave-6kf3"
server_key: "f1695fd5b5b905f0d152255f693a8294"
server_key_iv: "8c0eb09308bd9af8fd159b2144c36c5c"
server_key_version: "encryptionkey@0"
I20260924 19:36:45.041137 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:45.042129 3806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:45.042305 3791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:45.042409 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "b8c0c894ab844c3fb6047a8c7e089928"
format_stamp: "Formatted at 2026-09-24 19:36:45 on dist-test-slave-6kf3"
server_key: "f1695fd5b5b905f0d152255f693a8294"
server_key_iv: "8c0eb09308bd9af8fd159b2144c36c5c"
server_key_version: "encryptionkey@0"
I20260924 19:36:45.042477 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:45.064594 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:45.066351 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:45.066486 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:45.066700 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:45.066996 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:45.067036 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:45.067066 3791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:45.067087 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:45.072422 3791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36357
I20260924 19:36:45.072474 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36357 every 8 connection(s)
I20260924 19:36:45.072824 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:45.081046 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:45.073445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39727 (local address 127.0.98.62:36943)
0924 19:36:45.073488 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:45.073491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:45.073663 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:36:45.074140 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:45.074141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:45.074150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:45.074208 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:45.075076 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.075552 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.076525 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.076706 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.076751 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:45.078717 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:45.078718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:45.078719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:45.078726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:45.079811 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:45.080219 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:45.080220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:45.080221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:45.080278 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:45.080557 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:45.080557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:45.080557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:45.080656 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:45.080700 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:45.080907 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:36:45.080975 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:45.081204 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:45.073421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36943 (local address 127.0.98.3:39727)
0924 19:36:45.073560 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:45.073596 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:45.074062 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:45.074263 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:45.074266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:45.074810 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:45.075012 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:45.075017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.075609 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.075610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:45.076453 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:45.076456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.076760 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.076761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:45.076855 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:45.077354 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:45.077362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:45.078625 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:45.079873 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:45.079879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:45.079881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:45.080153 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:45.080344 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:45.080345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:45.080345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:45.080506 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:45.080698 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:45.080701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:45.080814 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 19:36:45.080889 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:36:45.081014 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 19:36:45.081677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3791
I20260924 19:36:45.081697 3920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36943
I20260924 19:36:45.081776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:45.081782 3920 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:45.081972 3920 heartbeater.cc:511] Master 127.0.98.62:36943 requested a full tablet report, sending...
I20260924 19:36:45.081984 392 external_mini_cluster.cc:1428] Setting key db4375ff9f932fdafb780f754310a8be
I20260924 19:36:45.082336 3460 ts_manager.cc:194] Registered new tserver with Master: b8c0c894ab844c3fb6047a8c7e089928 (127.0.98.3:36357)
I20260924 19:36:45.082710 3460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39727
I20260924 19:36:45.082783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278605, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278605, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:45.103011 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:45.097017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.0.98.62:36943)
0924 19:36:45.097090 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:45.097092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:45.097159 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:36:45.097213 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:45.097213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:45.097223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:45.097276 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:45.097769 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.098242 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.098894 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:45.099063 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:45.099105 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:45.101124 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:45.101126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:45.101126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:45.101132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:45.102186 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:45.102453 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:45.102454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:45.102454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:45.102498 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:45.102628 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:45.102628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:45.102629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:45.102756 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:45.102822 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:45.102890 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:36:45.102945 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 19:36:45.103255 3460 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56022
I20260924 19:36:45.103621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3523
I20260924 19:36:45.107497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3657
I20260924 19:36:45.110944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3791
I20260924 19:36:45.114491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3429
2026-09-24T19:36:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1711 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3932](info): set up 2 sockets
Sep 24 19:36:45 dist-test-slave-6kf3 krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260924 19:36:46.063659 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:46.172778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:46.252866 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:46.253091 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:46.253121 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:46.254369 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:46.254420 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:46.254438 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:46.254455 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:46.254469 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:46.255995 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:46.256251 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:46.256457 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:46.261019 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.261029 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:46.261255 3945 server_base.cc:1034] running on GCE node
W20260924 19:36:46.261127 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:46.261559 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:46.261845 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:46.262986 3945 hybrid_clock.cc:648] HybridClock initialized: now 1790278606262977 us; error 27 us; skew 500 ppm
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:46.265412 3945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:46.265830 3945 webserver.cc:460] Webserver started at http://127.0.98.62:35385/ using document root <none> and password file <none>
I20260924 19:36:46.266028 3945 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:46.266076 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:46.266202 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:46.267292 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "72160f498fc041e1b73652c12748594a"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "d6402d3259ff3ad416ad1df1e2ff7151"
server_key_iv: "3748f965bc68ed03ffddc891d12c0813"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.267632 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "72160f498fc041e1b73652c12748594a"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "d6402d3259ff3ad416ad1df1e2ff7151"
server_key_iv: "3748f965bc68ed03ffddc891d12c0813"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.268906 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:46.269550 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.269691 3945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:46.269773 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "72160f498fc041e1b73652c12748594a"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "d6402d3259ff3ad416ad1df1e2ff7151"
server_key_iv: "3748f965bc68ed03ffddc891d12c0813"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.269831 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:46.286123 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:46.288159 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:46.288298 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:46.292078 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45465
I20260924 19:36:46.292119 4012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45465 every 8 connection(s)
I20260924 19:36:46.292459 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:46.293256 4013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:46.295536 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a: Bootstrap starting.
I20260924 19:36:46.296165 4013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:46.296466 4013 log.cc:824] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:46.297084 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a: No bootstrap required, opened a new log
I20260924 19:36:46.297192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 3945
I20260924 19:36:46.297292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:46.297462 392 external_mini_cluster.cc:1428] Setting key fc6a071873d510fe3c8737dbc8d55b7b
I20260924 19:36:46.298889 4013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } }
I20260924 19:36:46.299032 4013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:46.299062 4013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72160f498fc041e1b73652c12748594a, State: Initialized, Role: FOLLOWER
I20260924 19:36:46.299192 4013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } }
I20260924 19:36:46.299265 4013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:46.299297 4013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:46.299345 4013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:46.300292 4013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } }
I20260924 19:36:46.300398 4013 leader_election.cc:302] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72160f498fc041e1b73652c12748594a; no voters:
I20260924 19:36:46.300545 4013 leader_election.cc:288] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:46.300586 4018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:46.300688 4018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [term 1 LEADER]: Becoming Leader. State: Replica: 72160f498fc041e1b73652c12748594a, State: Running, Role: LEADER
I20260924 19:36:46.300768 4018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } }
I20260924 19:36:46.300801 4013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:46.301144 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72160f498fc041e1b73652c12748594a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } } }
I20260924 19:36:46.301151 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72160f498fc041e1b73652c12748594a. Latest consensus state: current_term: 1 leader_uuid: "72160f498fc041e1b73652c12748594a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72160f498fc041e1b73652c12748594a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45465 } } }
I20260924 19:36:46.301234 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:46.301259 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:46.301436 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:46.301972 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:46.304005 4024 catalog_manager.cc:1347] Generated new cluster ID: 69104688f52c415c947926ff6bd8f40a
I20260924 19:36:46.304050 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:46.308702 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.297932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.98.62:45465)
0924 19:36:46.298115 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.298123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.298148 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:36:46.298856 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.298862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.298909 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.299032 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.299707 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.300786 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.301838 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.302454 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.302549 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.305217 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.305221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.305221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.305233 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.307043 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.307427 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.307430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.307431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.307536 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.307727 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.307728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.307728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.308025 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.308166 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.308273 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.308506 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260924 19:36:46.318286 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:46.318792 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:46.323467 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72160f498fc041e1b73652c12748594a: Generated new TSK 0
I20260924 19:36:46.323645 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:46.348376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:46.427734 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:46.427963 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:46.427996 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:46.429136 4039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:46.429171 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:46.429196 4039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:46.430629 4039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:46.430819 4039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:46.430999 4039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:46.435813 4047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.435850 4044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.435822 4045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:46.436077 4039 server_base.cc:1034] running on GCE node
I20260924 19:36:46.436292 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:46.436507 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:46.437641 4039 hybrid_clock.cc:648] HybridClock initialized: now 1790278606437627 us; error 30 us; skew 500 ppm
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:46.439992 4039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:46.440404 4039 webserver.cc:460] Webserver started at http://127.0.98.1:44385/ using document root <none> and password file <none>
I20260924 19:36:46.440616 4039 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:46.440659 4039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:46.440773 4039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:46.441758 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f2575be88c64dec9512acf2e0456e3f"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "402506c2da1a6028339646109ca21a0d"
server_key_iv: "f4670ab28389968489b2fefaf48c16aa"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.442077 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f2575be88c64dec9512acf2e0456e3f"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "402506c2da1a6028339646109ca21a0d"
server_key_iv: "f4670ab28389968489b2fefaf48c16aa"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.443238 4039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:46.443852 4054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.444008 4039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:46.444092 4039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "3f2575be88c64dec9512acf2e0456e3f"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "402506c2da1a6028339646109ca21a0d"
server_key_iv: "f4670ab28389968489b2fefaf48c16aa"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.444145 4039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:46.460870 4039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:46.462617 4039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:46.462724 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:46.462906 4039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:46.463153 4039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:46.463181 4039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.463201 4039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:46.463217 4039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.468552 4039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35193
I20260924 19:36:46.468593 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35193 every 8 connection(s)
I20260924 19:36:46.468889 4039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:46.472659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4039
I20260924 19:36:46.472743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:46.472914 392 external_mini_cluster.cc:1428] Setting key 6a0f2ce8f0304a0219bc6c3ab6883027
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:46.478232 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.469518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45599 (local address 127.0.98.62:45465)
0924 19:36:46.469594 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.469596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.469770 (+ 174us) server_negotiation.cc:407] Connection header received
0924 19:36:46.470672 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.470673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.470687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.470736 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.471311 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.471798 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.472972 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.473238 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.473303 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.475446 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.475448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.475449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.475456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.476932 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.477310 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.477312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.477312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.477371 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.477628 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.477629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.477629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.477737 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.477793 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.478056 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.478143 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:36:46.478402 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.469495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45465 (local address 127.0.98.1:45599)
0924 19:36:46.469655 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:46.469691 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:46.470590 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:46.470805 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:46.470809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:46.471058 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:46.471247 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.471251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.471863 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.471864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.472889 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.472892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.473324 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.473325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.473485 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.473970 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:46.473978 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:46.475354 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:46.477011 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.477017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.477020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.477231 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.477425 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.477426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.477426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.477568 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.477789 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:46.477795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:46.477939 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 19:36:46.478030 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:36:46.478185 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260924 19:36:46.478816 4168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45465
I20260924 19:36:46.478905 4168 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:46.479074 4168 heartbeater.cc:511] Master 127.0.98.62:45465 requested a full tablet report, sending...
I20260924 19:36:46.479562 3977 ts_manager.cc:194] Registered new tserver with Master: 3f2575be88c64dec9512acf2e0456e3f (127.0.98.1:35193)
I20260924 19:36:46.480201 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45599
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:46.495501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:46.573980 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:46.574193 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:46.574216 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:46.575361 4173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:46.575399 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:46.575425 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:46.576850 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:46.577036 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:46.577208 4173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:46.581969 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.582013 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.581996 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:46.582294 4173 server_base.cc:1034] running on GCE node
I20260924 19:36:46.582485 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:46.582697 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:46.583832 4173 hybrid_clock.cc:648] HybridClock initialized: now 1790278606583819 us; error 31 us; skew 500 ppm
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:46.586194 4173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:46.586639 4173 webserver.cc:460] Webserver started at http://127.0.98.2:39085/ using document root <none> and password file <none>
I20260924 19:36:46.586838 4173 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:46.586880 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:46.586990 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:46.588090 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a7c705fbcb94020a17aecf19d22b196"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "54f035b9eff81d6cb19c1649126a8db3"
server_key_iv: "b855925df1b15d9b00d96dcc0d6b0fb3"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.588415 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a7c705fbcb94020a17aecf19d22b196"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "54f035b9eff81d6cb19c1649126a8db3"
server_key_iv: "b855925df1b15d9b00d96dcc0d6b0fb3"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.589582 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:46.590248 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.590430 4173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:46.590512 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "2a7c705fbcb94020a17aecf19d22b196"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "54f035b9eff81d6cb19c1649126a8db3"
server_key_iv: "b855925df1b15d9b00d96dcc0d6b0fb3"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.590566 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:46.601765 4173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:46.603546 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:46.603677 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:46.603888 4173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:46.604190 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:46.604223 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.604250 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:46.604274 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.609516 4173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34759
I20260924 19:36:46.609596 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34759 every 8 connection(s)
I20260924 19:36:46.609860 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:46.618127 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.610490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55935 (local address 127.0.98.62:45465)
0924 19:36:46.610550 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.610552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.610718 (+ 166us) server_negotiation.cc:407] Connection header received
0924 19:36:46.611647 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.611649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.611661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.611711 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.612340 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.612832 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.613779 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.613970 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.614026 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.615898 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.615900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.615900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.615907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.616953 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.617283 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.617284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.617284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.617352 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.617550 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.617551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.617551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.617658 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.617714 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.617959 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.618038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:36:46.618281 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.610474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45465 (local address 127.0.98.2:55935)
0924 19:36:46.610605 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:46.610639 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:46.611541 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:46.611782 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:46.611785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:46.612063 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:46.612246 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.612250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.612909 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.612910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.613695 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.613697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.614034 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.614035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.614149 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.614574 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:46.614582 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:46.615816 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:46.617017 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.617024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.617026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.617221 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.617404 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.617404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.617405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.617498 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.617708 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:46.617714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:46.617847 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 19:36:46.617933 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:36:46.618076 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260924 19:36:46.618705 4302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45465
I20260924 19:36:46.618788 4302 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:46.618947 4302 heartbeater.cc:511] Master 127.0.98.62:45465 requested a full tablet report, sending...
I20260924 19:36:46.619285 3977 ts_manager.cc:194] Registered new tserver with Master: 2a7c705fbcb94020a17aecf19d22b196 (127.0.98.2:34759)
I20260924 19:36:46.619611 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55935
I20260924 19:36:46.619861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4173
I20260924 19:36:46.619985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:46.620174 392 external_mini_cluster.cc:1428] Setting key 7eda1f93c5d237469bb63c633840a799
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:46.642002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:46.722347 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:46.722561 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:46.722594 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:46.723758 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:46.723800 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:46.723835 4307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:46.725333 4307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:46.725574 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:46.725787 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:46.730554 4315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.730566 4313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:46.730566 4312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:46.730737 4307 server_base.cc:1034] running on GCE node
I20260924 19:36:46.730957 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:46.731151 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:46.732275 4307 hybrid_clock.cc:648] HybridClock initialized: now 1790278606732269 us; error 31 us; skew 500 ppm
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:46.734649 4307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:46.735100 4307 webserver.cc:460] Webserver started at http://127.0.98.3:43053/ using document root <none> and password file <none>
I20260924 19:36:46.735301 4307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:46.735347 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:46.735491 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:46.736577 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6c4db5ee4ee4281bcf2b07890217739"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "8877e8ea43eb2e0cdd9e40d27fe22dee"
server_key_iv: "e251b7bdf8967b949b99de2206b710b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.736904 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6c4db5ee4ee4281bcf2b07890217739"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "8877e8ea43eb2e0cdd9e40d27fe22dee"
server_key_iv: "e251b7bdf8967b949b99de2206b710b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.738106 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:46.738754 4322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.738920 4307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:46.739002 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "b6c4db5ee4ee4281bcf2b07890217739"
format_stamp: "Formatted at 2026-09-24 19:36:46 on dist-test-slave-6kf3"
server_key: "8877e8ea43eb2e0cdd9e40d27fe22dee"
server_key_iv: "e251b7bdf8967b949b99de2206b710b0"
server_key_version: "encryptionkey@0"
I20260924 19:36:46.739054 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:46.748370 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:46.750044 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:46.750155 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:46.750334 4307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:46.750618 4307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:46.750650 4307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.750671 4307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:46.750694 4307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:46.755985 4307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35775
I20260924 19:36:46.756031 4435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35775 every 8 connection(s)
I20260924 19:36:46.756332 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:46.764479 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.756958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42397 (local address 127.0.98.62:45465)
0924 19:36:46.757030 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.757032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.757190 (+ 158us) server_negotiation.cc:407] Connection header received
0924 19:36:46.758109 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.758110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.758120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.758158 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.758732 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.759212 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.760178 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.760349 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.760394 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.762219 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.762221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.762221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.762228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.763277 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.763618 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.763619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.763619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.763663 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.763840 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.763840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.763841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.763962 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.764034 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.764290 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.764388 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:36:46.764609 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.756935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45465 (local address 127.0.98.3:42397)
0924 19:36:46.757080 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:46.757115 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:46.758013 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:46.758234 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:46.758238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:46.758483 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:46.758672 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.758676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.759269 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.759270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.760122 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:46.760124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.760402 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.760403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:46.760494 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.760897 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:46.760904 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:46.762131 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:46.763361 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.763366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.763368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.763569 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.763703 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:46.763704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:46.763704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:46.763799 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:46.764029 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:46.764036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:46.764171 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:36:46.764258 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:36:46.764406 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260924 19:36:46.765025 4436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45465
I20260924 19:36:46.765112 4436 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:46.765267 4436 heartbeater.cc:511] Master 127.0.98.62:45465 requested a full tablet report, sending...
I20260924 19:36:46.765569 3977 ts_manager.cc:194] Registered new tserver with Master: b6c4db5ee4ee4281bcf2b07890217739 (127.0.98.3:35775)
I20260924 19:36:46.765900 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42397
I20260924 19:36:46.766215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4307
I20260924 19:36:46.766332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:46.766525 392 external_mini_cluster.cc:1428] Setting key a25dc2c069c10426f7b46af855c807c4
I20260924 19:36:46.767237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:46.788475 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.781454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.1:35193)
0924 19:36:46.781602 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.781607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.781615 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:46.781638 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.781640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.781653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.781715 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.782264 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.782970 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.783675 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.784021 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.784076 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.786078 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.786087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.786087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.786094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.787396 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.787685 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.787688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.787689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.787742 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.787905 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.787906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.787906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.788075 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.788217 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.788290 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.788363 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:46.795229 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:46.789665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.98.62:45465)
0924 19:36:46.789712 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:46.789714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:46.789726 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:36:46.789755 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:46.789755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:46.789765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:46.789818 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:46.790215 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.790697 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.791287 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:46.791464 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:46.791503 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:46.793390 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:46.793391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:46.793391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:46.793397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:46.794448 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.794706 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.794706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.794706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.794750 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:46.794863 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:46.794864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:46.794864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:46.794984 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:46.795047 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:46.795114 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:36:46.795165 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:36:46.795434 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48286
I20260924 19:36:46.795732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4039
I20260924 19:36:46.800446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4173
I20260924 19:36:46.804557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4307
I20260924 19:36:46.808635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 3945
2026-09-24T19:36:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1694 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[4448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[4448](info): set up 2 sockets
Sep 24 19:36:46 dist-test-slave-6kf3 krb5kdc[4448](info): commencing operation
krb5kdc: starting...
W20260924 19:36:47.755667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:47 dist-test-slave-6kf3 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:47.864197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:47.942557 4461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:47.942785 4461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:47.942807 4461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:47.943987 4461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:47.944032 4461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:47.944046 4461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:47.944056 4461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:47.944067 4461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:47.945583 4461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:47.945832 4461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:47.946041 4461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:47.950635 4469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:47.950803 4461 server_base.cc:1034] running on GCE node
W20260924 19:36:47.950651 4466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:47.950680 4467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:47.951100 4461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:47.951372 4461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:47.952512 4461 hybrid_clock.cc:648] HybridClock initialized: now 1790278607952493 us; error 34 us; skew 500 ppm
Sep 24 19:36:47 dist-test-slave-6kf3 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:47.961321 4461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:47.961678 4461 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:47.961730 4461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:47.961853 4461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:47.962924 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "5916d1751b06401792610ee5f1a848a4"
format_stamp: "Formatted at 2026-09-24 19:36:47 on dist-test-slave-6kf3"
server_key: "37b694887fccb41e04cf2493b99644ed"
server_key_iv: "896179a69c1b25bbee9e5a20c109b937"
server_key_version: "encryptionkey@0"
I20260924 19:36:47.963260 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "5916d1751b06401792610ee5f1a848a4"
format_stamp: "Formatted at 2026-09-24 19:36:47 on dist-test-slave-6kf3"
server_key: "37b694887fccb41e04cf2493b99644ed"
server_key_iv: "896179a69c1b25bbee9e5a20c109b937"
server_key_version: "encryptionkey@0"
I20260924 19:36:47.964511 4461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 19:36:47.965174 4475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:47.965324 4461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:47.965411 4461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "5916d1751b06401792610ee5f1a848a4"
format_stamp: "Formatted at 2026-09-24 19:36:47 on dist-test-slave-6kf3"
server_key: "37b694887fccb41e04cf2493b99644ed"
server_key_iv: "896179a69c1b25bbee9e5a20c109b937"
server_key_version: "encryptionkey@0"
I20260924 19:36:47.965471 4461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:47.978549 4461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:47.980619 4461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:47.980758 4461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:47.984414 4461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44737
I20260924 19:36:47.984436 4527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44737 every 8 connection(s)
I20260924 19:36:47.984704 4461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:47.985567 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:47.987763 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: Bootstrap starting.
I20260924 19:36:47.988338 4528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:47.988564 4528 log.cc:824] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:47.988675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4461
I20260924 19:36:47.988767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:47.988925 392 external_mini_cluster.cc:1428] Setting key 1d9cbea255e69e342ee50eb993bc6ec7
I20260924 19:36:47.989197 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: No bootstrap required, opened a new log
I20260924 19:36:47.991674 4528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260924 19:36:47.991814 4528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:47.991835 4528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5916d1751b06401792610ee5f1a848a4, State: Initialized, Role: FOLLOWER
I20260924 19:36:47.991972 4528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260924 19:36:47.992046 4528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:47.992069 4528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:47.992113 4528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:47.992740 4528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260924 19:36:47.992846 4528 leader_election.cc:302] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5916d1751b06401792610ee5f1a848a4; no voters:
I20260924 19:36:47.993001 4528 leader_election.cc:288] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:47.993043 4533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:47.993197 4533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [term 1 LEADER]: Becoming Leader. State: Replica: 5916d1751b06401792610ee5f1a848a4, State: Running, Role: LEADER
I20260924 19:36:47.993240 4528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:47.993371 4533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260924 19:36:47.993726 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5916d1751b06401792610ee5f1a848a4. Latest consensus state: current_term: 1 leader_uuid: "5916d1751b06401792610ee5f1a848a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } } }
I20260924 19:36:47.993796 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:47.994311 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5916d1751b06401792610ee5f1a848a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5916d1751b06401792610ee5f1a848a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } } }
I20260924 19:36:47.994383 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4 [sys.catalog]: This master's current role is: LEADER
W20260924 19:36:47.994994 4548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:47.995086 4548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:47.995198 4541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:47.995357 4541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:36:47 dist-test-slave-6kf3 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:47.997084 4541 catalog_manager.cc:1347] Generated new cluster ID: e409a4ce4cbc4ec597e4caf5db3a7843
I20260924 19:36:47.997141 4541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:47.999668 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:47.989388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.98.62:44737)
0924 19:36:47.989545 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:47.989554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:47.989579 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:36:47.990386 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:47.990394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:47.990453 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:47.990574 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:47.991336 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:47.992358 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:47.993363 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:47.993978 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:47.994060 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:47.996758 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:47.996763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:47.996763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:47.996775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:47.998349 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:47.998664 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:47.998667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:47.998667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:47.998748 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:47.998913 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:47.998914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:47.998914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:47.999142 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:47.999278 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:47.999370 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 19:36:47.999498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 19:36:48.016124 4541 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:48.016680 4541 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:48.023892 4541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: Generated new TSK 0
I20260924 19:36:48.024130 4541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:48.055094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:48.159896 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:48.160156 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:48.160179 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:48.161402 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:48.161451 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:48.161477 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:48.163005 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:48.163244 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:48.163462 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:36:48.168587 4554 server_base.cc:1034] running on GCE node
W20260924 19:36:48.168598 4562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.168630 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.168605 4560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.168984 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:48.169219 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:48.170351 4554 hybrid_clock.cc:648] HybridClock initialized: now 1790278608170343 us; error 26 us; skew 500 ppm
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:48.172729 4554 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:48.173064 4554 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:48.173131 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:48.173260 4554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:48.174270 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "9cfc7355749e9718813dbdd750c93495"
server_key_iv: "06946d13771922d9ecec62d79ab6a166"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.174599 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "9cfc7355749e9718813dbdd750c93495"
server_key_iv: "06946d13771922d9ecec62d79ab6a166"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.175736 4554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:48.176395 4568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.176527 4554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.176599 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "9cfc7355749e9718813dbdd750c93495"
server_key_iv: "06946d13771922d9ecec62d79ab6a166"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.176662 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:48.189035 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:48.191002 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:48.191133 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:48.191362 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:48.191671 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:48.191705 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.191735 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:48.191752 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.197394 4554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37225
I20260924 19:36:48.197435 4681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37225 every 8 connection(s)
I20260924 19:36:48.197706 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:48.200618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4554
I20260924 19:36:48.200711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:48.200863 392 external_mini_cluster.cc:1428] Setting key b6d6597f5eb4bd32ab1797fd7ae31ebf
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:48.207110 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.198391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43847 (local address 127.0.98.62:44737)
0924 19:36:48.198446 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.198449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.198646 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:36:48.199145 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.199146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.199158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.199220 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.200252 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.200836 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.201964 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.202183 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.202246 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.204398 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.204399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.204400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:48.204407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.205719 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.206200 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.206201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.206201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.206248 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.206499 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.206499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.206500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.206599 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.206647 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.206966 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.207034 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:36:48.207353 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.198354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.1:43847)
0924 19:36:48.198527 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.198563 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.199047 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.199298 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.199302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.199572 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:48.200137 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.200143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.200928 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.200929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.201883 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.201885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.202260 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.202261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.202397 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.202888 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:48.202896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:48.204293 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:48.205814 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.205822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.205825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.206125 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.206306 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.206307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.206307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.206442 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.206653 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:48.206659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:48.206830 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.206944 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.207113 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260924 19:36:48.207844 4682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260924 19:36:48.207960 4682 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:48.208112 4682 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:36:48.208597 4492 ts_manager.cc:194] Registered new tserver with Master: c178cf2fd48b48a3afbe8bfceb7a5ab8 (127.0.98.1:37225)
I20260924 19:36:48.209194 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:48.223723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:48.302642 4687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:48.302855 4687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:48.302889 4687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:48.304076 4687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:48.304121 4687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:48.304158 4687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:48.305641 4687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:48.305873 4687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:48.306073 4687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:48.310772 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.310773 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.310879 4687 server_base.cc:1034] running on GCE node
W20260924 19:36:48.310773 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.311196 4687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:48.311429 4687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:48.312577 4687 hybrid_clock.cc:648] HybridClock initialized: now 1790278608312562 us; error 27 us; skew 500 ppm
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:48.314987 4687 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:48.315290 4687 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:48.315335 4687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:48.315452 4687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:48.316511 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "7224f172ec5d41ad82d7b556b309c2a6"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "f003074133399136fa7d87945d174055"
server_key_iv: "b9cb212d996e48f3911e1165756ba75b"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.316835 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7224f172ec5d41ad82d7b556b309c2a6"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "f003074133399136fa7d87945d174055"
server_key_iv: "b9cb212d996e48f3911e1165756ba75b"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.318022 4687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:48.318686 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.318830 4687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:48.318903 4687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "7224f172ec5d41ad82d7b556b309c2a6"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "f003074133399136fa7d87945d174055"
server_key_iv: "b9cb212d996e48f3911e1165756ba75b"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.318969 4687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:48.329110 4687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:48.331228 4687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:48.331346 4687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:48.331570 4687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:48.331889 4687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:48.331921 4687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.331971 4687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:48.331990 4687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.337360 4687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41949
I20260924 19:36:48.337399 4814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41949 every 8 connection(s)
I20260924 19:36:48.337647 4687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:48.338228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4687
I20260924 19:36:48.338330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:48.338536 392 external_mini_cluster.cc:1428] Setting key da292d6b1913bb1cd057adbe773d6a7f
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:36:48.346959 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.338358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57253 (local address 127.0.98.62:44737)
0924 19:36:48.338415 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.338418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.338663 (+ 245us) server_negotiation.cc:407] Connection header received
0924 19:36:48.339260 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.339260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.339269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.339305 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.340567 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.341057 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.342067 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.342276 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.342333 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.344514 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.344516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.344516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:48.344524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.345753 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.346198 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.346199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.346200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.346250 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.346434 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.346434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.346434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.346531 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.346573 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.346822 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.346888 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:48.347177 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.338331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.2:57253)
0924 19:36:48.338527 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.338578 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.339169 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.339387 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.339393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.339778 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:48.340468 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.340475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.341148 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.341149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.341978 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.341981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.342349 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.342350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.342478 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.342946 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:48.342954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:48.344420 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:48.345844 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.345851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.345854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.346131 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.346300 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.346301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.346301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.346392 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.346572 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:48.346575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:48.346712 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.346792 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.346917 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260924 19:36:48.347622 4815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260924 19:36:48.347731 4815 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:48.347877 4815 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 19:36:48.348320 4492 ts_manager.cc:194] Registered new tserver with Master: 7224f172ec5d41ad82d7b556b309c2a6 (127.0.98.2:41949)
I20260924 19:36:48.348688 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:48.360932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:48.439270 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:48.439481 4820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:48.439503 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:48.440680 4820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:48.440722 4820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:48.440747 4820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:48.442226 4820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:48.442420 4820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:48.442613 4820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:48.447446 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.447443 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.447486 4820 server_base.cc:1034] running on GCE node
W20260924 19:36:48.447446 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.447845 4820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:48.448107 4820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:48.449254 4820 hybrid_clock.cc:648] HybridClock initialized: now 1790278608449241 us; error 29 us; skew 500 ppm
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:48.451632 4820 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:48.451925 4820 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:48.452006 4820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:48.452102 4820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:48.453102 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7d7ab6a369141f682824e2ae5c050e5"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "ef9d7af3c00e5129f41c1a6fc830c663"
server_key_iv: "1e9fa474bc42452ba925eaf581aa347c"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.453389 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d7ab6a369141f682824e2ae5c050e5"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "ef9d7af3c00e5129f41c1a6fc830c663"
server_key_iv: "1e9fa474bc42452ba925eaf581aa347c"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.454495 4820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:48.455137 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.455266 4820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:48.455344 4820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "d7d7ab6a369141f682824e2ae5c050e5"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "ef9d7af3c00e5129f41c1a6fc830c663"
server_key_iv: "1e9fa474bc42452ba925eaf581aa347c"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.455401 4820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:48.465955 4820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:48.468024 4820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:48.468155 4820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:48.468359 4820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:48.468658 4820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:48.468689 4820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.468719 4820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:48.468742 4820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.474006 4820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33451
I20260924 19:36:48.474040 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33451 every 8 connection(s)
I20260924 19:36:48.474287 4820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:48.475066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4820
I20260924 19:36:48.475167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:48.475356 392 external_mini_cluster.cc:1428] Setting key c5b750d9ea247b03de363045e21aec49
Sep 24 19:36:48 dist-test-slave-6kf3 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:48.482831 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.474935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38813 (local address 127.0.98.62:44737)
0924 19:36:48.474998 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.475000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.475185 (+ 185us) server_negotiation.cc:407] Connection header received
0924 19:36:48.475671 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.475672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.475684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.475751 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.476732 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.477234 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.478211 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.478409 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.478462 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.480514 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.480515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.480516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:48.480524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.481577 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.481942 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.481944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.481944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.481996 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.482228 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.482229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.482230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.482337 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.482391 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.482660 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.482739 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:36:48.482998 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.474905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.3:38813)
0924 19:36:48.475055 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.475096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.475572 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.475826 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.475830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.476140 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:48.476642 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.476648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.477316 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.477317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.478133 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.478135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.478477 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.478478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.478605 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.479019 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:48.479027 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:48.480415 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:48.481660 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.481667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.481669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.481879 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.482048 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:48.482048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:48.482049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:48.482174 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:48.482388 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:48.482395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:48.482542 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.482632 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.482786 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 19:36:48.483436 4948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260924 19:36:48.483520 4948 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:48.483645 4948 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
I20260924 19:36:48.483994 4492 ts_manager.cc:194] Registered new tserver with Master: d7d7ab6a369141f682824e2ae5c050e5 (127.0.98.3:33451)
I20260924 19:36:48.484369 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38813
I20260924 19:36:48.486760 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:36:48.493180 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.488018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.0.98.62:44737)
0924 19:36:48.488071 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.488073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.488150 (+ 77us) server_negotiation.cc:407] Connection header received
0924 19:36:48.488199 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.488199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.488207 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.488249 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.488733 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.489224 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.489888 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.490058 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.490117 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.491154 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.491154 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.491155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:48.491159 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.492246 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.492527 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.492528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.492528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.492571 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:48.492695 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:48.492696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:48.492696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:48.492878 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.492951 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.493025 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.493102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:36:48.494798 4492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:48.496367 4492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:48.502720 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33451 (local address 127.0.0.1:53158)
0924 19:36:48.498860 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.498913 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.498984 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.499232 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.499236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.499238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:48.499307 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.499309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500609 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.500610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.501644 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.501646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502409 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.502493 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502508 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.502584 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.502639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":31,"thread_start_us":51,"threads_started":1}
I20260924 19:36:48.502734 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.0.98.3:33451)
0924 19:36:48.498755 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.498760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.498970 (+ 210us) server_negotiation.cc:407] Connection header received
0924 19:36:48.499068 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.499116 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.499136 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.499213 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:48.499367 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500514 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.501724 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502329 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502399 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502456 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.502610 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.502652 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260924 19:36:48.503052 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.98.1:37225)
0924 19:36:48.498975 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.498979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.499256 (+ 277us) server_negotiation.cc:407] Connection header received
0924 19:36:48.499298 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.499300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.499315 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.499389 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:48.499531 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500619 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502012 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502536 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502602 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.502940 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.502978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":74,"threads_started":1}
I20260924 19:36:48.503054 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.0.98.2:41949)
0924 19:36:48.498891 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.498896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.499228 (+ 332us) server_negotiation.cc:407] Connection header received
0924 19:36:48.499545 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.499548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.499563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.499629 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:48.499827 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500558 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.501998 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502659 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502719 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502768 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.502943 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.502980 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":152,"threads_started":1}
I20260924 19:36:48.502889 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37225 (local address 127.0.0.1:55466)
0924 19:36:48.499065 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.499075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.499124 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.499396 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.499397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.499400 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:48.499449 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.499450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500708 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.500709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.501919 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.501921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502688 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.502742 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502745 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.502788 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.502824 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":74,"thread_start_us":54,"threads_started":1}
I20260924 19:36:48.503026 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.498733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41949 (local address 127.0.0.1:32996)
0924 19:36:48.499175 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.499181 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.499457 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.499723 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.499724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.499726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:48.499763 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.499764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.500656 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.500657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.501918 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.501921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.502802 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.502803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.502864 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.502869 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.502918 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.502961 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":70,"thread_start_us":69,"threads_started":1}
I20260924 19:36:48.503965 4616 tablet_service.cc:1467] Processing CreateTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 (DEFAULT_TABLE table=test-table [id=fa2f6a398fbf42498d71c9fbc52ef3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:48.503974 4882 tablet_service.cc:1467] Processing CreateTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 (DEFAULT_TABLE table=test-table [id=fa2f6a398fbf42498d71c9fbc52ef3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:48.504155 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 (DEFAULT_TABLE table=test-table [id=fa2f6a398fbf42498d71c9fbc52ef3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:48.504338 4882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a029c0a620e4c15a788f7cdc12880d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:48.504338 4616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a029c0a620e4c15a788f7cdc12880d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:48.504431 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a029c0a620e4c15a788f7cdc12880d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:48.506789 4962 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: Bootstrap starting.
I20260924 19:36:48.507377 4962 tablet_bootstrap.cc:654] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:48.507426 4963 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Bootstrap starting.
I20260924 19:36:48.508042 4963 tablet_bootstrap.cc:654] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:48.508096 4964 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Bootstrap starting.
I20260924 19:36:48.508134 4962 log.cc:824] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:48.508565 4963 log.cc:824] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:48.508659 4964 tablet_bootstrap.cc:654] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:48.508945 4962 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: No bootstrap required, opened a new log
I20260924 19:36:48.509002 4962 ts_tablet_manager.cc:1397] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:48.509130 4964 log.cc:824] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:48.509701 4964 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: No bootstrap required, opened a new log
I20260924 19:36:48.509752 4964 ts_tablet_manager.cc:1397] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:48.510414 4962 raft_consensus.cc:348] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.510556 4962 raft_consensus.cc:374] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:48.510596 4962 raft_consensus.cc:729] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178cf2fd48b48a3afbe8bfceb7a5ab8, State: Initialized, Role: FOLLOWER
I20260924 19:36:48.510712 4962 consensus_queue.cc:260] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.510973 4962 ts_tablet_manager.cc:1428] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:48.510998 4682 heartbeater.cc:503] Master 127.0.98.62:44737 was elected leader, sending a full tablet report...
I20260924 19:36:48.511356 4964 raft_consensus.cc:348] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.511520 4964 raft_consensus.cc:374] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:48.511564 4964 raft_consensus.cc:729] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7224f172ec5d41ad82d7b556b309c2a6, State: Initialized, Role: FOLLOWER
I20260924 19:36:48.511564 4963 tablet_bootstrap.cc:492] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: No bootstrap required, opened a new log
I20260924 19:36:48.511629 4963 ts_tablet_manager.cc:1397] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 19:36:48.511669 4964 consensus_queue.cc:260] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.511988 4815 heartbeater.cc:503] Master 127.0.98.62:44737 was elected leader, sending a full tablet report...
I20260924 19:36:48.511983 4964 ts_tablet_manager.cc:1428] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:36:48.513252 4963 raft_consensus.cc:348] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.513368 4963 raft_consensus.cc:374] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:48.513387 4963 raft_consensus.cc:729] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d7ab6a369141f682824e2ae5c050e5, State: Initialized, Role: FOLLOWER
I20260924 19:36:48.513487 4963 consensus_queue.cc:260] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.513708 4963 ts_tablet_manager.cc:1428] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:36:48.513723 4948 heartbeater.cc:503] Master 127.0.98.62:44737 was elected leader, sending a full tablet report...
W20260924 19:36:48.588150 4816 tablet.cc:2369] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:36:48.654214 4968 raft_consensus.cc:482] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:48.654346 4968 raft_consensus.cc:504] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.654729 4968 leader_election.cc:288] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7224f172ec5d41ad82d7b556b309c2a6 (127.0.98.2:41949), d7d7ab6a369141f682824e2ae5c050e5 (127.0.98.3:33451)
I20260924 19:36:48.657424 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.654921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33451 (local address 127.0.98.1:36271)
0924 19:36:48.654964 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.654976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.655035 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.655176 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.655177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.655178 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:48.655242 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.655244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.655833 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.655834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.656658 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.656660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.657180 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.657181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.657258 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.657261 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.657304 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.657344 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 19:36:48.657433 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.654975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36271 (local address 127.0.98.3:33451)
0924 19:36:48.655027 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.655029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.655042 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:36:48.655093 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.655094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.655106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.655162 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:48.655299 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.655761 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.656739 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.657115 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.657174 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.657183 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.657315 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.657350 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:48.657621 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.655144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41265 (local address 127.0.98.2:41949)
0924 19:36:48.655186 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.655188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.655293 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:36:48.655346 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.655347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.655358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.655400 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:48.655549 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.656048 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.656926 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.657285 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.657343 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.657352 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.657513 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.657553 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:36:48.657811 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.654966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41949 (local address 127.0.98.1:41265)
0924 19:36:48.655235 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.655244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.655292 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.655402 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.655402 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.655404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:48.655494 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.655496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.656110 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.656111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.656843 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:48.656845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.657350 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.657351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:48.657429 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.657432 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.657697 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.657737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260924 19:36:48.657786 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a029c0a620e4c15a788f7cdc12880d3" candidate_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d7ab6a369141f682824e2ae5c050e5" is_pre_election: true
I20260924 19:36:48.657943 4902 raft_consensus.cc:2454] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178cf2fd48b48a3afbe8bfceb7a5ab8 in term 0.
I20260924 19:36:48.658134 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a029c0a620e4c15a788f7cdc12880d3" candidate_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7224f172ec5d41ad82d7b556b309c2a6" is_pre_election: true
I20260924 19:36:48.658169 4570 leader_election.cc:302] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178cf2fd48b48a3afbe8bfceb7a5ab8, d7d7ab6a369141f682824e2ae5c050e5; no voters:
I20260924 19:36:48.658285 4769 raft_consensus.cc:2454] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178cf2fd48b48a3afbe8bfceb7a5ab8 in term 0.
I20260924 19:36:48.658327 4968 raft_consensus.cc:2781] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:48.658398 4968 raft_consensus.cc:482] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:48.658433 4968 raft_consensus.cc:3037] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:48.659175 4968 raft_consensus.cc:504] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.659289 4968 leader_election.cc:288] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 7224f172ec5d41ad82d7b556b309c2a6 (127.0.98.2:41949), d7d7ab6a369141f682824e2ae5c050e5 (127.0.98.3:33451)
I20260924 19:36:48.659430 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a029c0a620e4c15a788f7cdc12880d3" candidate_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7224f172ec5d41ad82d7b556b309c2a6"
I20260924 19:36:48.659479 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a029c0a620e4c15a788f7cdc12880d3" candidate_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d7ab6a369141f682824e2ae5c050e5"
I20260924 19:36:48.659515 4769 raft_consensus.cc:3037] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:48.659540 4902 raft_consensus.cc:3037] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:48.660247 4769 raft_consensus.cc:2454] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178cf2fd48b48a3afbe8bfceb7a5ab8 in term 1.
I20260924 19:36:48.660277 4902 raft_consensus.cc:2454] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178cf2fd48b48a3afbe8bfceb7a5ab8 in term 1.
I20260924 19:36:48.660398 4570 leader_election.cc:302] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178cf2fd48b48a3afbe8bfceb7a5ab8, d7d7ab6a369141f682824e2ae5c050e5; no voters:
I20260924 19:36:48.660534 4968 raft_consensus.cc:2781] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:48.660693 4968 raft_consensus.cc:686] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 1 LEADER]: Becoming Leader. State: Replica: c178cf2fd48b48a3afbe8bfceb7a5ab8, State: Running, Role: LEADER
I20260924 19:36:48.660781 4968 consensus_queue.cc:237] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } }
I20260924 19:36:48.661598 4492 catalog_manager.cc:5495] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178cf2fd48b48a3afbe8bfceb7a5ab8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178cf2fd48b48a3afbe8bfceb7a5ab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:48.682272 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.680149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.98.1:37225)
0924 19:36:48.680191 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.680193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.680230 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:36:48.680291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.680292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.680313 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.680370 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:48.680491 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.680949 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.681653 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:48.681834 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:48.681880 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:48.682002 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:48.682114 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:48.682168 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.682182 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.682199 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:36:48.683945 4769 raft_consensus.cc:1261] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 1 FOLLOWER]: Refusing update from remote peer c178cf2fd48b48a3afbe8bfceb7a5ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:48.683919 4902 raft_consensus.cc:1261] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 1 FOLLOWER]: Refusing update from remote peer c178cf2fd48b48a3afbe8bfceb7a5ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:48.684279 4972 consensus_queue.cc:1035] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d7ab6a369141f682824e2ae5c050e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:48.684405 4968 consensus_queue.cc:1035] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7224f172ec5d41ad82d7b556b309c2a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:48.686519 4975 mvcc.cc:205] Tried to move back new op lower bound from 7332981181167030272 to 7332981181075451904. Current Snapshot: MvccSnapshot[applied={T|T < 7332981181167030272}]
I20260924 19:36:48.687398 4977 mvcc.cc:205] Tried to move back new op lower bound from 7332981181167030272 to 7332981181075451904. Current Snapshot: MvccSnapshot[applied={T|T < 7332981181167030272}]
I20260924 19:36:48.689150 4492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:48.689234 4492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:48.689890 4492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5916d1751b06401792610ee5f1a848a4: Sending DeleteTablet for 3 replicas of tablet 9a029c0a620e4c15a788f7cdc12880d3
I20260924 19:36:48.690227 4882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53158
I20260924 19:36:48.690237 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32996
I20260924 19:36:48.690232 4616 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a029c0a620e4c15a788f7cdc12880d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55466
I20260924 19:36:48.690439 4986 tablet_replica.cc:330] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8: stopping tablet replica
I20260924 19:36:48.690505 4984 tablet_replica.cc:330] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: stopping tablet replica
I20260924 19:36:48.690533 4986 raft_consensus.cc:2229] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:48.690586 4984 raft_consensus.cc:2229] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:48.690652 4986 raft_consensus.cc:2258] T 9a029c0a620e4c15a788f7cdc12880d3 P c178cf2fd48b48a3afbe8bfceb7a5ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:48.690670 4984 raft_consensus.cc:2258] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:48.690860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4554
I20260924 19:36:48.690920 4985 tablet_replica.cc:330] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: stopping tablet replica
I20260924 19:36:48.691008 4985 raft_consensus.cc:2229] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:48.691098 4985 raft_consensus.cc:2258] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:48.691530 4985 ts_tablet_manager.cc:1905] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:48.692250 4984 ts_tablet_manager.cc:1905] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:48.693331 4985 ts_tablet_manager.cc:1918] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:48.693401 4985 log.cc:1196] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/9a029c0a620e4c15a788f7cdc12880d3
I20260924 19:36:48.693661 4984 ts_tablet_manager.cc:1918] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:48.693712 4985 ts_tablet_manager.cc:1939] T 9a029c0a620e4c15a788f7cdc12880d3 P 7224f172ec5d41ad82d7b556b309c2a6: Deleting consensus metadata
I20260924 19:36:48.693763 4984 log.cc:1196] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/9a029c0a620e4c15a788f7cdc12880d3
I20260924 19:36:48.693992 4984 ts_tablet_manager.cc:1939] T 9a029c0a620e4c15a788f7cdc12880d3 P d7d7ab6a369141f682824e2ae5c050e5: Deleting consensus metadata
I20260924 19:36:48.694123 4479 catalog_manager.cc:4841] TS 7224f172ec5d41ad82d7b556b309c2a6 (127.0.98.2:41949): tablet 9a029c0a620e4c15a788f7cdc12880d3 (table test-table [id=fa2f6a398fbf42498d71c9fbc52ef3c7]) successfully deleted
I20260924 19:36:48.694353 4477 catalog_manager.cc:4841] TS d7d7ab6a369141f682824e2ae5c050e5 (127.0.98.3:33451): tablet 9a029c0a620e4c15a788f7cdc12880d3 (table test-table [id=fa2f6a398fbf42498d71c9fbc52ef3c7]) successfully deleted
W20260924 19:36:48.696435 4478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37225 (error 108)
I20260924 19:36:48.696635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4687
I20260924 19:36:48.697181 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.696759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37225 (local address 127.0.0.1:55488)
0924 19:36:48.697104 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.697148 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320}
W20260924 19:36:48.700526 4478 catalog_manager.cc:4584] TS c178cf2fd48b48a3afbe8bfceb7a5ab8 (127.0.98.1:37225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a029c0a620e4c15a788f7cdc12880d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37225: connect: Connection refused (error 111)
I20260924 19:36:48.700816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4820
I20260924 19:36:48.704510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4461
2026-09-24T19:36:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1895 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:48Z Disabled control of system clock
I20260924 19:36:48.729326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34771
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:36:48.806921 4991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:48.807132 4991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:48.807152 4991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:48.808276 4991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:48.808316 4991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:48.808328 4991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:48.808339 4991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:48.808355 4991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:48.809643 4991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:48.809841 4991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:48.810086 4991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:48.814579 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.814670 4996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.814610 4997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.814802 4991 server_base.cc:1034] running on GCE node
I20260924 19:36:48.815017 4991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:48.815308 4991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:48.816437 4991 hybrid_clock.cc:648] HybridClock initialized: now 1790278608816427 us; error 26 us; skew 500 ppm
I20260924 19:36:48.817628 4991 webserver.cc:460] Webserver started at http://127.0.98.62:46001/ using document root <none> and password file <none>
I20260924 19:36:48.817852 4991 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:48.817898 4991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:48.818022 4991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:48.819062 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "50c974fa619b42a19447ac28e888453a"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "401fc97522e92b2949f8622ef2fb745a"
server_key_iv: "c33156601ec447b1462a8ff07c8a5275"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.819392 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "50c974fa619b42a19447ac28e888453a"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "401fc97522e92b2949f8622ef2fb745a"
server_key_iv: "c33156601ec447b1462a8ff07c8a5275"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.820607 4991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:48.821233 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.821365 4991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:48.821431 4991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "50c974fa619b42a19447ac28e888453a"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "401fc97522e92b2949f8622ef2fb745a"
server_key_iv: "c33156601ec447b1462a8ff07c8a5275"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.821497 4991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:48.824687 4991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:48.824923 4991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:48.825044 4991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:48.828652 4991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34771
I20260924 19:36:48.828684 5057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34771 every 8 connection(s)
I20260924 19:36:48.828994 4991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:48.829612 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:48.832326 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: Bootstrap starting.
I20260924 19:36:48.832857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 4991
I20260924 19:36:48.832895 5058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:48.832942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:48.833117 392 external_mini_cluster.cc:1428] Setting key 6a35e35f08c3010363d24804d8d15e70
I20260924 19:36:48.833153 5058 log.cc:824] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:48.833817 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: No bootstrap required, opened a new log
I20260924 19:36:48.835537 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.833630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.0.98.62:34771)
0924 19:36:48.833821 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.833828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.833851 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:36:48.834293 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.834301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.834340 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.834465 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.835136 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.835140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.835140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:48.835145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.835187 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:48.835197 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.835265 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.835315 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.835389 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260924 19:36:48.835659 5058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } }
I20260924 19:36:48.835821 5058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:48.835856 5058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c974fa619b42a19447ac28e888453a, State: Initialized, Role: FOLLOWER
I20260924 19:36:48.836050 5058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } }
I20260924 19:36:48.836129 5058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:48.836159 5058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:48.836205 5058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:48.836969 5058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } }
I20260924 19:36:48.837059 5058 leader_election.cc:302] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c974fa619b42a19447ac28e888453a; no voters:
I20260924 19:36:48.837182 5058 leader_election.cc:288] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:48.837229 5063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:48.837383 5063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [term 1 LEADER]: Becoming Leader. State: Replica: 50c974fa619b42a19447ac28e888453a, State: Running, Role: LEADER
I20260924 19:36:48.837399 5058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:48.837527 5063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } }
I20260924 19:36:48.837858 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c974fa619b42a19447ac28e888453a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } } }
I20260924 19:36:48.837949 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:48.837852 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c974fa619b42a19447ac28e888453a. Latest consensus state: current_term: 1 leader_uuid: "50c974fa619b42a19447ac28e888453a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c974fa619b42a19447ac28e888453a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34771 } } }
I20260924 19:36:48.838100 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:48.838404 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:48.838843 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:48.840240 5071 catalog_manager.cc:1347] Generated new cluster ID: 2d980019b3a1400893340f1fbc8e5677
I20260924 19:36:48.840291 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:48.857167 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:48.857332 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:48.861794 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: Generated new TSK 0
I20260924 19:36:48.861969 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:36:48.864429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:36:48.945703 5082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:48.945916 5082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:48.945946 5082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:48.947122 5082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:48.947162 5082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:48.947196 5082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:48.948549 5082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:48.948773 5082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:48.949054 5082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:48.953778 5087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.953749 5088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:48.953904 5090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:48.954165 5082 server_base.cc:1034] running on GCE node
I20260924 19:36:48.954361 5082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:48.954592 5082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:48.955746 5082 hybrid_clock.cc:648] HybridClock initialized: now 1790278608955727 us; error 30 us; skew 500 ppm
I20260924 19:36:48.957031 5082 webserver.cc:460] Webserver started at http://127.0.98.1:36127/ using document root <none> and password file <none>
I20260924 19:36:48.957273 5082 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:48.957320 5082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:48.957448 5082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:48.958609 5082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "a24ef636862e4851be34083b932701b4"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "4180875cb84cc71a8042f0250a04be71"
server_key_iv: "77a70a15d91d885e472e741b589d6a55"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.958945 5082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a24ef636862e4851be34083b932701b4"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "4180875cb84cc71a8042f0250a04be71"
server_key_iv: "77a70a15d91d885e472e741b589d6a55"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.960222 5082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:48.960888 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.961046 5082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:48.961124 5082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "a24ef636862e4851be34083b932701b4"
format_stamp: "Formatted at 2026-09-24 19:36:48 on dist-test-slave-6kf3"
server_key: "4180875cb84cc71a8042f0250a04be71"
server_key_iv: "77a70a15d91d885e472e741b589d6a55"
server_key_version: "encryptionkey@0"
I20260924 19:36:48.961185 5082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:48.964406 5082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:48.964655 5082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:48.964771 5082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:48.964953 5082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:48.965219 5082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:48.965247 5082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.965272 5082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:48.965296 5082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:48.970495 5082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37777
I20260924 19:36:48.970532 5209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37777 every 8 connection(s)
I20260924 19:36:48.970831 5082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:48.973536 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.971483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38235 (local address 127.0.98.62:34771)
0924 19:36:48.971543 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:48.971545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:48.971762 (+ 217us) server_negotiation.cc:407] Connection header received
0924 19:36:48.972278 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:48.972279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:48.972284 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:48.972341 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:48.973348 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:48.973349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:48.973349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:48.973353 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:48.973366 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:48.973369 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:48.973413 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:48.973469 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 19:36:48.973481 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:36:48.973726 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:48.971457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34771 (local address 127.0.98.1:38235)
0924 19:36:48.971637 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:48.971683 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:48.972197 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:48.972397 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:48.972400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:48.972408 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:48.973219 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:48.973219 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:48.973247 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:48.973275 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:48.973417 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:48.973420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:48.973422 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:36:48.973473 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:36:48.973519 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260924 19:36:48.974196 5210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34771
I20260924 19:36:48.974282 5210 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:48.974422 5210 heartbeater.cc:511] Master 127.0.98.62:34771 requested a full tablet report, sending...
I20260924 19:36:48.974889 5022 ts_manager.cc:194] Registered new tserver with Master: a24ef636862e4851be34083b932701b4 (127.0.98.1:37777)
I20260924 19:36:48.980630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5082
I20260924 19:36:48.980715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:48.980863 392 external_mini_cluster.cc:1428] Setting key 6baaad769266ed30aa68da0f202e945b
I20260924 19:36:48.981695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:36:49.061137 5213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:49.061340 5213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:49.061371 5213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:49.062541 5213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:49.062577 5213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:49.062608 5213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:49.064041 5213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:49.064271 5213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:49.064582 5213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:49.069267 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.069273 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.069474 5213 server_base.cc:1034] running on GCE node
W20260924 19:36:49.069263 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.069753 5213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:49.069976 5213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:49.071111 5213 hybrid_clock.cc:648] HybridClock initialized: now 1790278609071093 us; error 33 us; skew 500 ppm
I20260924 19:36:49.072352 5213 webserver.cc:460] Webserver started at http://127.0.98.2:46283/ using document root <none> and password file <none>
I20260924 19:36:49.072576 5213 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:49.072618 5213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:49.072713 5213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:49.073711 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "24fbc3e35cdf48edad6ca0addcaf41ea"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "94568b5fe2a0dcceae053ed77336922b"
server_key_iv: "f9da5c23889e86274f2e8c51a61c2513"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.074008 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24fbc3e35cdf48edad6ca0addcaf41ea"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "94568b5fe2a0dcceae053ed77336922b"
server_key_iv: "f9da5c23889e86274f2e8c51a61c2513"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.075165 5213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:36:49.075838 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.075999 5213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.076077 5213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "24fbc3e35cdf48edad6ca0addcaf41ea"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "94568b5fe2a0dcceae053ed77336922b"
server_key_iv: "f9da5c23889e86274f2e8c51a61c2513"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.076146 5213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:49.079322 5213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:49.079561 5213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:49.079687 5213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:49.079872 5213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:49.080201 5213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:49.080235 5213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.080261 5213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:49.080286 5213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.085481 5213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44479
I20260924 19:36:49.085515 5340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44479 every 8 connection(s)
I20260924 19:36:49.085825 5213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:49.088636 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.086487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49367 (local address 127.0.98.62:34771)
0924 19:36:49.086556 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.086558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.086763 (+ 205us) server_negotiation.cc:407] Connection header received
0924 19:36:49.087253 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.087254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.087261 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.087312 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.088405 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.088406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.088407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.088411 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.088426 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.088430 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.088478 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.088553 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.088567 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:36:49.088815 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.086472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34771 (local address 127.0.98.2:49367)
0924 19:36:49.086641 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.086674 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.087155 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.087381 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.087385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.087395 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.088262 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.088262 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.088292 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.088319 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.088481 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.088486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.088489 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.088557 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.088617 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 19:36:49.089250 5341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34771
I20260924 19:36:49.089322 5341 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:49.089435 5341 heartbeater.cc:511] Master 127.0.98.62:34771 requested a full tablet report, sending...
I20260924 19:36:49.089740 5022 ts_manager.cc:194] Registered new tserver with Master: 24fbc3e35cdf48edad6ca0addcaf41ea (127.0.98.2:44479)
I20260924 19:36:49.095798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5213
I20260924 19:36:49.095873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:49.096060 392 external_mini_cluster.cc:1428] Setting key be7ca175c88af6e4842f14fd591cb801
I20260924 19:36:49.096851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:36:49.175206 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:49.175407 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:49.175427 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:49.176613 5344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:49.176654 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:49.176682 5344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:49.178020 5344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:49.178231 5344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:49.178534 5344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:36:49.183288 5344 server_base.cc:1034] running on GCE node
W20260924 19:36:49.183259 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.183243 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.183246 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.183609 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:49.183825 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:49.184954 5344 hybrid_clock.cc:648] HybridClock initialized: now 1790278609184939 us; error 31 us; skew 500 ppm
I20260924 19:36:49.186122 5344 webserver.cc:460] Webserver started at http://127.0.98.3:38379/ using document root <none> and password file <none>
I20260924 19:36:49.186342 5344 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:49.186388 5344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:49.186509 5344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:49.187536 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "68044155c10347ae96198796db663943"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "866a4e8e2bc8af983558b152aa14e553"
server_key_iv: "ddfe6f59df1fbadd28b5dbb6b4b70416"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.187848 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68044155c10347ae96198796db663943"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "866a4e8e2bc8af983558b152aa14e553"
server_key_iv: "ddfe6f59df1fbadd28b5dbb6b4b70416"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.189081 5344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:36:49.189720 5358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.189867 5344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:49.189939 5344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "68044155c10347ae96198796db663943"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "866a4e8e2bc8af983558b152aa14e553"
server_key_iv: "ddfe6f59df1fbadd28b5dbb6b4b70416"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.190018 5344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:49.193271 5344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:49.193516 5344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:49.193627 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:49.193804 5344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:49.194123 5344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:49.194164 5344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.194190 5344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:49.194217 5344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.199473 5344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44393
I20260924 19:36:49.199522 5471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44393 every 8 connection(s)
I20260924 19:36:49.199816 5344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:49.200805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5344
I20260924 19:36:49.200897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:49.201053 392 external_mini_cluster.cc:1428] Setting key ac4064a401e285b21f729b78803ecf79
I20260924 19:36:49.202677 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.200534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52397 (local address 127.0.98.62:34771)
0924 19:36:49.200589 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.200591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.200804 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:36:49.201350 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.201351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.201357 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.201421 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.202514 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.202515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.202515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.202518 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.202530 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.202534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.202570 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.202617 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.202627 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:36:49.202853 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.200515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34771 (local address 127.0.98.3:52397)
0924 19:36:49.200683 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.200721 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.201245 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.201494 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.201499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.201508 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.202387 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.202388 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.202416 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.202445 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.202572 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.202574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.202577 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.202623 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.202668 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 19:36:49.203217 5472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34771
I20260924 19:36:49.203299 5472 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:49.203433 5472 heartbeater.cc:511] Master 127.0.98.62:34771 requested a full tablet report, sending...
I20260924 19:36:49.203709 5022 ts_manager.cc:194] Registered new tserver with Master: 68044155c10347ae96198796db663943 (127.0.98.3:44393)
I20260924 19:36:49.212548 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:36:49.214725 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.213811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:34771)
0924 19:36:49.213858 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.213860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.213942 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:36:49.214001 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.214002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.214006 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.214049 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.214573 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.214574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.214574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.214576 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.214585 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.214587 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.214619 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.214648 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.214659 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:36:49.216136 5022 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:49.217599 5022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:49.220937 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.219489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37777 (local address 127.0.0.1:57124)
0924 19:36:49.219622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.219657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.219713 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.220008 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.220014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.220022 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.220459 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.220460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.220477 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.220511 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.220803 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.220810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.220813 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.220856 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.220869 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 19:36:49.221074 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.219844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44393 (local address 127.0.0.1:48130)
0924 19:36:49.219953 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.219960 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.219991 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.220423 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.220424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.220427 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.220765 (+ 338us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.220766 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.220773 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.220783 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.220958 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.220959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.220960 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.221014 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.221025 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":59,"threads_started":1}
I20260924 19:36:49.221233 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.219618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.0.98.1:37777)
0924 19:36:49.219820 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.219825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.219835 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:49.219860 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.219862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.219872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.220204 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.220671 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.220675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.220676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.220679 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.220712 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.220727 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.221113 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.221139 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.221155 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260924 19:36:49.221408 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.220063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.0.98.3:44393)
0924 19:36:49.220298 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.220302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.220309 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:49.220330 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.220332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.220340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.220568 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.220832 (+ 264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.220836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.220836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.220839 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.220866 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.220879 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.220958 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.221314 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.221330 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260924 19:36:49.221547 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.219697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44479 (local address 127.0.0.1:50672)
0924 19:36:49.219806 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.219814 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.219846 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.220151 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.220152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.220154 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.221213 (+ 1059us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.221214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.221219 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.221230 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.221416 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.221417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.221417 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.221462 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.221481 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260924 19:36:49.221760 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.219804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.0.98.2:44479)
0924 19:36:49.220017 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.220022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.220032 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:49.220057 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.220059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.220067 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.220138 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.221290 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.221300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.221301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.221305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.221340 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.221363 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.221421 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.221656 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.221679 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260924 19:36:49.221875 5144 tablet_service.cc:1467] Processing CreateTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 (DEFAULT_TABLE table=test-table [id=be55cac59760406d9b1108ae1c1abd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:49.222158 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c9c1e30b5e401e9ceed40fc6515287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.222278 5275 tablet_service.cc:1467] Processing CreateTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 (DEFAULT_TABLE table=test-table [id=be55cac59760406d9b1108ae1c1abd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:49.222559 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c9c1e30b5e401e9ceed40fc6515287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.224570 5406 tablet_service.cc:1467] Processing CreateTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 (DEFAULT_TABLE table=test-table [id=be55cac59760406d9b1108ae1c1abd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:49.224933 5406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c9c1e30b5e401e9ceed40fc6515287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.224983 5486 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: Bootstrap starting.
I20260924 19:36:49.225207 5487 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: Bootstrap starting.
I20260924 19:36:49.225600 5486 tablet_bootstrap.cc:654] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.225762 5487 tablet_bootstrap.cc:654] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.225907 5486 log.cc:824] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.226017 5487 log.cc:824] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.226640 5486 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: No bootstrap required, opened a new log
I20260924 19:36:49.226711 5486 ts_tablet_manager.cc:1397] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:49.226810 5487 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: No bootstrap required, opened a new log
I20260924 19:36:49.226903 5487 ts_tablet_manager.cc:1397] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:49.227744 5490 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Bootstrap starting.
I20260924 19:36:49.228384 5490 tablet_bootstrap.cc:654] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.228652 5490 log.cc:824] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.229269 5490 tablet_bootstrap.cc:492] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: No bootstrap required, opened a new log
I20260924 19:36:49.229322 5490 ts_tablet_manager.cc:1397] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:49.229339 5486 raft_consensus.cc:348] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.229529 5486 raft_consensus.cc:374] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.229571 5486 raft_consensus.cc:729] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24ef636862e4851be34083b932701b4, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.229594 5487 raft_consensus.cc:348] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.229725 5487 raft_consensus.cc:374] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.229697 5486 consensus_queue.cc:260] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.229763 5487 raft_consensus.cc:729] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fbc3e35cdf48edad6ca0addcaf41ea, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.229934 5486 ts_tablet_manager.cc:1428] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:49.229904 5487 consensus_queue.cc:260] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.230057 5210 heartbeater.cc:503] Master 127.0.98.62:34771 was elected leader, sending a full tablet report...
I20260924 19:36:49.230139 5487 ts_tablet_manager.cc:1428] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:49.230149 5341 heartbeater.cc:503] Master 127.0.98.62:34771 was elected leader, sending a full tablet report...
I20260924 19:36:49.231848 5490 raft_consensus.cc:348] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.232008 5490 raft_consensus.cc:374] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.232044 5490 raft_consensus.cc:729] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68044155c10347ae96198796db663943, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.232143 5490 consensus_queue.cc:260] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.232371 5490 ts_tablet_manager.cc:1428] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:49.232394 5472 heartbeater.cc:503] Master 127.0.98.62:34771 was elected leader, sending a full tablet report...
I20260924 19:36:49.251955 5492 raft_consensus.cc:482] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:49.252144 5492 raft_consensus.cc:504] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.252480 5492 leader_election.cc:288] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24fbc3e35cdf48edad6ca0addcaf41ea (127.0.98.2:44479), 68044155c10347ae96198796db663943 (127.0.98.3:44393)
I20260924 19:36:49.253700 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.252663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44479 (local address 127.0.98.1:57023)
0924 19:36:49.252721 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.252733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.252775 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.252932 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.252933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.252935 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.253395 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.253395 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.253402 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.253413 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.253578 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.253579 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.253580 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.253617 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.253631 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:36:49.253700 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.252760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57023 (local address 127.0.98.2:44479)
0924 19:36:49.252807 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.252809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.252819 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:49.252847 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.252847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.252853 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.252906 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.253505 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.253507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.253507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.253511 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.253527 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.253531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.253577 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.253627 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.253639 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:36:49.253823 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.252836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55225 (local address 127.0.98.3:44393)
0924 19:36:49.252876 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.252878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.253000 (+ 122us) server_negotiation.cc:407] Connection header received
0924 19:36:49.253029 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.253030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.253035 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.253089 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.253629 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.253630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.253630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.253635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.253651 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.253655 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.253703 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.253748 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.253761 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:49.253834 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.252757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44393 (local address 127.0.98.1:55225)
0924 19:36:49.252936 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.252944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.252990 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.253094 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.253095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.253098 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:49.253532 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:36:49.253532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:49.253539 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:36:49.253552 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:49.253705 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:49.253706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:49.253706 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:36:49.253749 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:36:49.253760 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260924 19:36:49.254047 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c9c1e30b5e401e9ceed40fc6515287" candidate_uuid: "a24ef636862e4851be34083b932701b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" is_pre_election: true
I20260924 19:36:49.254186 5295 raft_consensus.cc:2454] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24ef636862e4851be34083b932701b4 in term 0.
I20260924 19:36:49.254295 5426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c9c1e30b5e401e9ceed40fc6515287" candidate_uuid: "a24ef636862e4851be34083b932701b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68044155c10347ae96198796db663943" is_pre_election: true
I20260924 19:36:49.254411 5099 leader_election.cc:302] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fbc3e35cdf48edad6ca0addcaf41ea, a24ef636862e4851be34083b932701b4; no voters:
I20260924 19:36:49.254442 5426 raft_consensus.cc:2454] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24ef636862e4851be34083b932701b4 in term 0.
I20260924 19:36:49.254541 5492 raft_consensus.cc:2781] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:49.254590 5492 raft_consensus.cc:482] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:49.254609 5492 raft_consensus.cc:3037] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.255283 5492 raft_consensus.cc:504] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.255388 5492 leader_election.cc:288] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [CANDIDATE]: Term 1 election: Requested vote from peers 24fbc3e35cdf48edad6ca0addcaf41ea (127.0.98.2:44479), 68044155c10347ae96198796db663943 (127.0.98.3:44393)
I20260924 19:36:49.255542 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c9c1e30b5e401e9ceed40fc6515287" candidate_uuid: "a24ef636862e4851be34083b932701b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea"
I20260924 19:36:49.255566 5426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c9c1e30b5e401e9ceed40fc6515287" candidate_uuid: "a24ef636862e4851be34083b932701b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68044155c10347ae96198796db663943"
I20260924 19:36:49.255622 5426 raft_consensus.cc:3037] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.255625 5295 raft_consensus.cc:3037] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.256325 5426 raft_consensus.cc:2454] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24ef636862e4851be34083b932701b4 in term 1.
I20260924 19:36:49.256362 5295 raft_consensus.cc:2454] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24ef636862e4851be34083b932701b4 in term 1.
I20260924 19:36:49.256476 5099 leader_election.cc:302] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68044155c10347ae96198796db663943, a24ef636862e4851be34083b932701b4; no voters:
I20260924 19:36:49.256556 5492 raft_consensus.cc:2781] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:49.256698 5492 raft_consensus.cc:686] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 1 LEADER]: Becoming Leader. State: Replica: a24ef636862e4851be34083b932701b4, State: Running, Role: LEADER
I20260924 19:36:49.256775 5492 consensus_queue.cc:237] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260924 19:36:49.257579 5022 catalog_manager.cc:5495] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a24ef636862e4851be34083b932701b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a24ef636862e4851be34083b932701b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24ef636862e4851be34083b932701b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: UNKNOWN } } }
I20260924 19:36:49.270367 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.269509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.98.1:37777)
0924 19:36:49.269567 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.269569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.269577 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:49.269604 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.269605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.269615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.269661 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.270168 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.270170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.270170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.270174 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.270189 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.270193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.270240 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.270283 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.270296 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:36:49.271921 5426 raft_consensus.cc:1261] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 1 FOLLOWER]: Refusing update from remote peer a24ef636862e4851be34083b932701b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:49.271970 5295 raft_consensus.cc:1261] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 1 FOLLOWER]: Refusing update from remote peer a24ef636862e4851be34083b932701b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:49.272270 5496 consensus_queue.cc:1035] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fbc3e35cdf48edad6ca0addcaf41ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:49.272374 5492 consensus_queue.cc:1035] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68044155c10347ae96198796db663943" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:49.274794 5501 mvcc.cc:205] Tried to move back new op lower bound from 7332981183575633920 to 7332981183516463104. Current Snapshot: MvccSnapshot[applied={T|T < 7332981183575633920}]
I20260924 19:36:49.275388 5500 mvcc.cc:205] Tried to move back new op lower bound from 7332981183575633920 to 7332981183516463104. Current Snapshot: MvccSnapshot[applied={T|T < 7332981183575633920}]
I20260924 19:36:49.276680 5021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:49.276757 5021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:49.277421 5021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50c974fa619b42a19447ac28e888453a: Sending DeleteTablet for 3 replicas of tablet b7c9c1e30b5e401e9ceed40fc6515287
I20260924 19:36:49.277681 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:49 UTC) from {username='slave'} at 127.0.0.1:57124
I20260924 19:36:49.277747 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:49 UTC) from {username='slave'} at 127.0.0.1:50672
I20260924 19:36:49.277748 5406 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c9c1e30b5e401e9ceed40fc6515287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:49 UTC) from {username='slave'} at 127.0.0.1:48130
I20260924 19:36:49.277942 5508 tablet_replica.cc:330] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4: stopping tablet replica
I20260924 19:36:49.278033 5508 raft_consensus.cc:2229] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:49.278110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5082
I20260924 19:36:49.278118 5508 raft_consensus.cc:2258] T b7c9c1e30b5e401e9ceed40fc6515287 P a24ef636862e4851be34083b932701b4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:36:49.282545 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37777 (error 108)
I20260924 19:36:49.282615 5510 tablet_replica.cc:330] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: stopping tablet replica
I20260924 19:36:49.282717 5510 raft_consensus.cc:2229] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:49.282811 5509 tablet_replica.cc:330] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea: stopping tablet replica
I20260924 19:36:49.282830 5510 raft_consensus.cc:2258] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:49.282902 5509 raft_consensus.cc:2229] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:49.282990 5509 raft_consensus.cc:2258] T b7c9c1e30b5e401e9ceed40fc6515287 P 24fbc3e35cdf48edad6ca0addcaf41ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:49.283063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5213
I20260924 19:36:49.283133 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.283017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37777 (local address 127.0.0.1:57148)
0924 19:36:49.283065 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.283101 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:36:49.283263 5008 catalog_manager.cc:4584] TS a24ef636862e4851be34083b932701b4 (127.0.98.1:37777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7c9c1e30b5e401e9ceed40fc6515287: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37777: connect: Connection refused (error 111)
I20260924 19:36:49.283314 5510 ts_tablet_manager.cc:1905] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:49.284747 5510 ts_tablet_manager.cc:1918] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:49.284812 5510 log.cc:1196] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/b7c9c1e30b5e401e9ceed40fc6515287
I20260924 19:36:49.285027 5510 ts_tablet_manager.cc:1939] T b7c9c1e30b5e401e9ceed40fc6515287 P 68044155c10347ae96198796db663943: Deleting consensus metadata
I20260924 19:36:49.285385 5008 catalog_manager.cc:4841] TS 68044155c10347ae96198796db663943 (127.0.98.3:44393): tablet b7c9c1e30b5e401e9ceed40fc6515287 (table test-table [id=be55cac59760406d9b1108ae1c1abd68]) successfully deleted
I20260924 19:36:49.287336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5344
I20260924 19:36:49.287349 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.287245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44479 (local address 127.0.0.1:50682)
0924 19:36:49.287286 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.287318 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:36:49.291651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 4991
2026-09-24T19:36:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (586 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:36:49.309762 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:36:49.310745 392 webserver.cc:460] Webserver started at https://127.0.0.1:37891/ using document root <none> and password file <none>
I20260924 19:36:49.310822 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:36:49.310945 392 webserver.cc:460] Webserver started at http://127.0.0.1:43149/ using document root <none> and password file <none>
2026-09-24T19:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:49Z Disabled control of system clock
I20260924 19:36:49.317700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45589
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/testchainca.cert with env {}
W20260924 19:36:49.394495 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:49.394698 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:49.394726 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:49.395864 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:49.395905 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:49.395920 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:49.395959 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:49.395984 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:36:49.396004 5517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:36:49.396026 5517 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37891/jwks.json
I20260924 19:36:49.397531 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:49.397769 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:49.397959 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:49.402415 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.402437 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.402437 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.402457 5517 server_base.cc:1034] running on GCE node
I20260924 19:36:49.402781 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:49.403067 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:49.404202 5517 hybrid_clock.cc:648] HybridClock initialized: now 1790278609404185 us; error 29 us; skew 500 ppm
I20260924 19:36:49.405342 5517 webserver.cc:460] Webserver started at http://127.0.98.62:41861/ using document root <none> and password file <none>
I20260924 19:36:49.405552 5517 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:49.405601 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:49.405719 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:49.406745 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "c485cb57c85b44adbc8dd6ad3f567928"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "dda97ec97056c461032c5d3f85517d0c"
server_key_iv: "d59875cd4647d849c67a675ff6395eb0"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.407068 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "c485cb57c85b44adbc8dd6ad3f567928"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "dda97ec97056c461032c5d3f85517d0c"
server_key_iv: "d59875cd4647d849c67a675ff6395eb0"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.408200 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:49.408831 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.408981 5517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:49.409060 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "c485cb57c85b44adbc8dd6ad3f567928"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "dda97ec97056c461032c5d3f85517d0c"
server_key_iv: "d59875cd4647d849c67a675ff6395eb0"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.409117 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:49.450090 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:49.450410 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:49.450542 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:49.454109 5517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45589
I20260924 19:36:49.454142 5583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45589 every 8 connection(s)
I20260924 19:36:49.454453 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:49.455348 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.457571 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928: Bootstrap starting.
I20260924 19:36:49.458092 5584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.458325 5584 log.cc:824] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.458951 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928: No bootstrap required, opened a new log
I20260924 19:36:49.460531 5584 raft_consensus.cc:348] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } }
I20260924 19:36:49.460668 5584 raft_consensus.cc:374] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.460688 5584 raft_consensus.cc:729] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c485cb57c85b44adbc8dd6ad3f567928, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.460796 5584 consensus_queue.cc:260] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } }
I20260924 19:36:49.460867 5584 raft_consensus.cc:388] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:49.460888 5584 raft_consensus.cc:482] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:49.460919 5584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.461351 5584 raft_consensus.cc:504] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } }
I20260924 19:36:49.461459 5584 leader_election.cc:302] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c485cb57c85b44adbc8dd6ad3f567928; no voters:
I20260924 19:36:49.461563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5517
I20260924 19:36:49.461634 5584 leader_election.cc:288] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:49.461658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:49.461705 5587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:49.461838 392 external_mini_cluster.cc:1428] Setting key f78354e35a7cee4b29067715af7b5726
I20260924 19:36:49.461849 5587 raft_consensus.cc:686] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [term 1 LEADER]: Becoming Leader. State: Replica: c485cb57c85b44adbc8dd6ad3f567928, State: Running, Role: LEADER
I20260924 19:36:49.461887 5584 sys_catalog.cc:564] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:49.461958 5587 consensus_queue.cc:237] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } }
I20260924 19:36:49.462294 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c485cb57c85b44adbc8dd6ad3f567928. Latest consensus state: current_term: 1 leader_uuid: "c485cb57c85b44adbc8dd6ad3f567928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } } }
I20260924 19:36:49.462370 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.462373 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c485cb57c85b44adbc8dd6ad3f567928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485cb57c85b44adbc8dd6ad3f567928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45589 } } }
I20260924 19:36:49.462642 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.462765 5596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:49.463294 5596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:49.464807 5596 catalog_manager.cc:1347] Generated new cluster ID: 7e49dff57d4b49788fb20048768f9912
I20260924 19:36:49.464859 5596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:49.467191 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.462592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.98.62:45589)
0924 19:36:49.462779 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.462788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.462819 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:36:49.462933 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.462941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.463013 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.463412 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.463534 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.464692 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.465505 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.465913 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.466019 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.466629 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.466632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.466633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.466789 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.466820 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.466831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.466883 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.466952 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.467028 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260924 19:36:49.468748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:36:49.485157 5596 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:49.485683 5596 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:49.496558 5596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c485cb57c85b44adbc8dd6ad3f567928: Generated new TSK 0
I20260924 19:36:49.496737 5596 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:36:49.526646 5614 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:36:49.526751 5614 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:36:49.527030 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.504767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.0.98.62:45589)
0924 19:36:49.504804 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.504806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.504941 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:36:49.505041 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.505042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.505054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.505109 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:36:49.505249 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.505730 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.506477 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.506654 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.506705 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.506888 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:36:49.526646 (+ 19758us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:36:49.526955 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:36:49.527091 5598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:36:49.527226 5613 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:49.504728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45589 (local address 127.0.0.1:50410)
0924 19:36:49.504881 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.504890 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.504950 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.505119 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.505120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.505122 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:36:49.505189 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.505191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.505801 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.505802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.506397 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.506398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.506707 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.506711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.506775 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.506797 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:36:49.526780 (+ 19983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:36:49.526891 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:36:49.527587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5517
2026-09-24T19:36:49Z chronyd exiting
I20260924 19:36:49.563017 392 test_util.cc:175] -----------------------------------------------
I20260924 19:36:49.563133 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278584646480-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (254 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:36:49.564142 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:36:49.564954 392 webserver.cc:460] Webserver started at https://127.0.0.1:39247/ using document root <none> and password file <none>
I20260924 19:36:49.565013 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:36:49.565160 392 webserver.cc:460] Webserver started at http://127.0.0.1:38121/ using document root <none> and password file <none>
2026-09-24T19:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:49Z Disabled control of system clock
I20260924 19:36:49.572455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38073
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38073
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/testchainca.cert with env {}
W20260924 19:36:49.649714 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:49.649922 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:49.649943 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:49.651111 5620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:49.651154 5620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:49.651165 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:49.651185 5620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:49.651209 5620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:36:49.651232 5620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:36:49.651252 5620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39247/jwks.json
I20260924 19:36:49.652814 5620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:49.653054 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:49.653251 5620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:49.657774 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.657882 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.657989 5620 server_base.cc:1034] running on GCE node
W20260924 19:36:49.657750 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.658244 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:49.658492 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:49.659618 5620 hybrid_clock.cc:648] HybridClock initialized: now 1790278609659602 us; error 34 us; skew 500 ppm
I20260924 19:36:49.661015 5620 webserver.cc:460] Webserver started at http://127.0.98.62:34305/ using document root <none> and password file <none>
I20260924 19:36:49.661245 5620 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:49.661289 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:49.661391 5620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:49.662431 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "07059a657f5f4aef8a03b318e4754acf"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "ba42e23be39c93035284dd3ede6ec4f4"
server_key_iv: "e9de7c09c99e1c161847e7e6f9293523"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.662763 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "07059a657f5f4aef8a03b318e4754acf"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "ba42e23be39c93035284dd3ede6ec4f4"
server_key_iv: "e9de7c09c99e1c161847e7e6f9293523"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.664017 5620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:49.664710 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.664848 5620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:49.664929 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "07059a657f5f4aef8a03b318e4754acf"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "ba42e23be39c93035284dd3ede6ec4f4"
server_key_iv: "e9de7c09c99e1c161847e7e6f9293523"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.664988 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:49.689491 5620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:49.689777 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:49.689877 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:49.693461 5620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38073
I20260924 19:36:49.693516 5686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38073 every 8 connection(s)
I20260924 19:36:49.693814 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:49.694717 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.696312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5620
I20260924 19:36:49.696415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:49.696595 392 external_mini_cluster.cc:1428] Setting key 9068c811c9b6b92978aef714f444eede
I20260924 19:36:49.697365 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf: Bootstrap starting.
I20260924 19:36:49.697898 5687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.698203 5687 log.cc:824] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.698807 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf: No bootstrap required, opened a new log
I20260924 19:36:49.700973 5687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260924 19:36:49.701129 5687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.701162 5687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07059a657f5f4aef8a03b318e4754acf, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.701288 5687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260924 19:36:49.701354 5687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:49.701382 5687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:49.701427 5687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.701934 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.697145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.0.98.62:38073)
0924 19:36:49.697307 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.697315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.697339 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:36:49.697779 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.697787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.697849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.697976 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.698188 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.699087 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.700166 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.700550 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.700630 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.701279 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.701282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.701283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.701470 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.701512 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.701526 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.701578 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.701683 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.701748 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 19:36:49.702015 5687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260924 19:36:49.702112 5687 leader_election.cc:302] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07059a657f5f4aef8a03b318e4754acf; no voters:
I20260924 19:36:49.702404 5687 leader_election.cc:288] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:49.702450 5692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:49.702564 5692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [term 1 LEADER]: Becoming Leader. State: Replica: 07059a657f5f4aef8a03b318e4754acf, State: Running, Role: LEADER
I20260924 19:36:49.702646 5687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:49.702636 5692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260924 19:36:49.702702 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:36:49.702951 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07059a657f5f4aef8a03b318e4754acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } } }
I20260924 19:36:49.703008 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.703118 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07059a657f5f4aef8a03b318e4754acf. Latest consensus state: current_term: 1 leader_uuid: "07059a657f5f4aef8a03b318e4754acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07059a657f5f4aef8a03b318e4754acf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } } }
I20260924 19:36:49.703161 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.703473 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:49.703974 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:49.705415 5697 catalog_manager.cc:1347] Generated new cluster ID: 44751b376b3b4fce88fe70c7b490a802
I20260924 19:36:49.705454 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:49.737733 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:49.738257 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:49.746574 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07059a657f5f4aef8a03b318e4754acf: Generated new TSK 0
I20260924 19:36:49.746748 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:36:49.772140 5717 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:36:49.772223 5717 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:36:49.772538 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.758805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.0.98.62:38073)
0924 19:36:49.758843 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.758845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.758958 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:36:49.759031 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.759032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.759044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.759104 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:36:49.759248 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.759735 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.760531 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.760732 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.760786 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.760931 (+ 145us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:36:49.772317 (+ 11386us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:36:49.772459 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:36:49.772593 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 19:36:49.772619 5716 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:49.758766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38073 (local address 127.0.0.1:43528)
0924 19:36:49.758899 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.758908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.758965 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.759108 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.759109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.759111 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:36:49.759170 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.759172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.759809 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.759810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.760459 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.760462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.760789 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.760793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.760849 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.760854 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:36:49.772430 (+ 11576us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:36:49.772537 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260924 19:36:49.772876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5620
2026-09-24T19:36:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:36:49.809314 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:36:49.809950 392 webserver.cc:460] Webserver started at https://127.0.0.1:35507/ using document root <none> and password file <none>
I20260924 19:36:49.809993 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:36:49.810097 392 webserver.cc:460] Webserver started at http://127.0.0.1:38887/ using document root <none> and password file <none>
2026-09-24T19:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:49Z Disabled control of system clock
I20260924 19:36:49.816709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38209
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38209
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:36:49.893865 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:49.894068 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:49.894096 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:49.895234 5723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:49.895275 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:49.895292 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:49.895308 5723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:49.895323 5723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:36:49.895339 5723 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:36:49.895361 5723 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35507/jwks.json
I20260924 19:36:49.896785 5723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:49.897024 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:49.897267 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:49.901911 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:49.901875 5729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.902038 5723 server_base.cc:1034] running on GCE node
W20260924 19:36:49.901883 5728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:49.902403 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:49.902688 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:49.903852 5723 hybrid_clock.cc:648] HybridClock initialized: now 1790278609903834 us; error 41 us; skew 500 ppm
I20260924 19:36:49.905229 5723 webserver.cc:460] Webserver started at http://127.0.98.62:40409/ using document root <none> and password file <none>
I20260924 19:36:49.905436 5723 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:49.905484 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:49.905609 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:49.906620 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "3f5e6775e25240dc85b89653822f5a3b"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "2f46cbb6e723dfb4030a0a92d6cca6ca"
server_key_iv: "19763c029a458d9743ab2fd4664b5293"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.906960 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f5e6775e25240dc85b89653822f5a3b"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "2f46cbb6e723dfb4030a0a92d6cca6ca"
server_key_iv: "19763c029a458d9743ab2fd4664b5293"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.908142 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:49.908798 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:49.908962 5723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:49.909039 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "3f5e6775e25240dc85b89653822f5a3b"
format_stamp: "Formatted at 2026-09-24 19:36:49 on dist-test-slave-6kf3"
server_key: "2f46cbb6e723dfb4030a0a92d6cca6ca"
server_key_iv: "19763c029a458d9743ab2fd4664b5293"
server_key_version: "encryptionkey@0"
I20260924 19:36:49.909096 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:49.920189 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:49.920430 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:49.920542 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:49.923877 5723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38209
I20260924 19:36:49.923916 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38209 every 8 connection(s)
I20260924 19:36:49.924222 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:49.924782 5790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:49.927181 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b: Bootstrap starting.
I20260924 19:36:49.927698 5790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:49.927996 5790 log.cc:824] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:49.928586 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b: No bootstrap required, opened a new log
I20260924 19:36:49.930177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5723
I20260924 19:36:49.930094 5790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260924 19:36:49.930220 5790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:49.930253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:49.930250 5790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5e6775e25240dc85b89653822f5a3b, State: Initialized, Role: FOLLOWER
I20260924 19:36:49.930375 5790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260924 19:36:49.930411 392 external_mini_cluster.cc:1428] Setting key 056ce19ccd09f59e292020b8fce68ce0
I20260924 19:36:49.930449 5790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:49.930480 5790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:49.930527 5790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:49.931183 5790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260924 19:36:49.931290 5790 leader_election.cc:302] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f5e6775e25240dc85b89653822f5a3b; no voters:
I20260924 19:36:49.931432 5790 leader_election.cc:288] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:49.931499 5795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:49.931679 5795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [term 1 LEADER]: Becoming Leader. State: Replica: 3f5e6775e25240dc85b89653822f5a3b, State: Running, Role: LEADER
I20260924 19:36:49.931665 5790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:49.931981 5795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260924 19:36:49.932372 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f5e6775e25240dc85b89653822f5a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } } }
I20260924 19:36:49.932456 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.932585 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f5e6775e25240dc85b89653822f5a3b. Latest consensus state: current_term: 1 leader_uuid: "3f5e6775e25240dc85b89653822f5a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f5e6775e25240dc85b89653822f5a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } } }
I20260924 19:36:49.932641 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:49.933106 5804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:49.933723 5804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:49.935585 5804 catalog_manager.cc:1347] Generated new cluster ID: 4be4ce33bc8044a0945e0868118a7b55
I20260924 19:36:49.935642 5804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:49.936379 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.930816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.98.62:38209)
0924 19:36:49.931041 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.931050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.931089 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:36:49.931209 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.931215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.931304 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.931501 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:49.931862 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.933485 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.934290 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.934737 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.934843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.935731 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:49.935735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:49.935735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:49.935896 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:49.935954 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:49.935969 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:49.936006 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:49.936117 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:36:49.936197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260924 19:36:49.940308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:36:49.970549 5804 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:49.971153 5804 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:49.974764 5804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f5e6775e25240dc85b89653822f5a3b: Generated new TSK 0
I20260924 19:36:49.974928 5804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:36:49.987613 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:49.984289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.98.62:38209)
0924 19:36:49.984330 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:49.984332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:49.984391 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:36:49.984467 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:49.984467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:49.984477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:49.984534 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:36:49.984660 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.985126 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.986262 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.986446 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.986492 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.986657 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:36:49.987422 (+ 765us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:36:49.987545 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:36:49.987682 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 19:36:49.987686 5819 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:49.984205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38209 (local address 127.0.0.1:49944)
0924 19:36:49.984325 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:49.984336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:49.984400 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:49.984545 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:49.984546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:49.984548 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:36:49.984605 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.984608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.985204 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.985205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.986183 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:49.986187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:49.986506 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:49.986513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:49.986573 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:49.986579 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:36:49.987520 (+ 941us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:36:49.987604 (+ 84us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38209: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260924 19:36:49.988057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5723
E20260924 19:36:50.000465 5820 webserver.cc:533] Webserver: sslize failed:
E20260924 19:36:50.000543 5820 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T19:36:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:50Z Disabled control of system clock
I20260924 19:36:50.032922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37767
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37767
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/testchainca.cert with env {}
W20260924 19:36:50.110936 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:50.111136 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:50.111156 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:50.112335 5824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:50.112376 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:50.112387 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:50.112401 5824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:50.112411 5824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:36:50.112423 5824 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:36:50.112447 5824 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:36:50.113921 5824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:50.114159 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:50.114355 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:50.118875 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:50.118870 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:50.118875 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:50.119220 5824 server_base.cc:1034] running on GCE node
I20260924 19:36:50.119448 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:50.119724 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:50.120857 5824 hybrid_clock.cc:648] HybridClock initialized: now 1790278610120836 us; error 33 us; skew 500 ppm
I20260924 19:36:50.121978 5824 webserver.cc:460] Webserver started at http://127.0.98.62:42837/ using document root <none> and password file <none>
I20260924 19:36:50.122193 5824 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:50.122241 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:50.122361 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:50.123415 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "fffeb67254264278ae7ba509bdb19e39"
format_stamp: "Formatted at 2026-09-24 19:36:50 on dist-test-slave-6kf3"
server_key: "072344f9599d5db4ca819cb227eb2e26"
server_key_iv: "d2bdd9847bf8f09f8f47ff490392419a"
server_key_version: "encryptionkey@0"
I20260924 19:36:50.123751 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "fffeb67254264278ae7ba509bdb19e39"
format_stamp: "Formatted at 2026-09-24 19:36:50 on dist-test-slave-6kf3"
server_key: "072344f9599d5db4ca819cb227eb2e26"
server_key_iv: "d2bdd9847bf8f09f8f47ff490392419a"
server_key_version: "encryptionkey@0"
I20260924 19:36:50.125013 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:36:50.125679 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:50.125833 5824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:50.125909 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "fffeb67254264278ae7ba509bdb19e39"
format_stamp: "Formatted at 2026-09-24 19:36:50 on dist-test-slave-6kf3"
server_key: "072344f9599d5db4ca819cb227eb2e26"
server_key_iv: "d2bdd9847bf8f09f8f47ff490392419a"
server_key_version: "encryptionkey@0"
I20260924 19:36:50.125978 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:50.145745 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:50.146061 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:50.146193 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:50.149994 5824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37767
I20260924 19:36:50.150040 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37767 every 8 connection(s)
I20260924 19:36:50.150359 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:50.151221 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:50.153429 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39: Bootstrap starting.
I20260924 19:36:50.153960 5891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:50.154202 5891 log.cc:824] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:50.154788 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39: No bootstrap required, opened a new log
I20260924 19:36:50.156363 5891 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260924 19:36:50.156518 5891 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:50.156562 5891 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffeb67254264278ae7ba509bdb19e39, State: Initialized, Role: FOLLOWER
I20260924 19:36:50.156677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5824
I20260924 19:36:50.156710 5891 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260924 19:36:50.156769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:50.156790 5891 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:50.156824 5891 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:50.156869 5891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:50.156947 392 external_mini_cluster.cc:1428] Setting key 2d096ed373b7779ee0abb6980dc1040c
I20260924 19:36:50.157439 5891 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260924 19:36:50.157546 5891 leader_election.cc:302] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffeb67254264278ae7ba509bdb19e39; no voters:
I20260924 19:36:50.157693 5891 leader_election.cc:288] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:50.157729 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:50.157851 5896 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [term 1 LEADER]: Becoming Leader. State: Replica: fffeb67254264278ae7ba509bdb19e39, State: Running, Role: LEADER
I20260924 19:36:50.157930 5891 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:50.157935 5896 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260924 19:36:50.158356 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffeb67254264278ae7ba509bdb19e39. Latest consensus state: current_term: 1 leader_uuid: "fffeb67254264278ae7ba509bdb19e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } } }
I20260924 19:36:50.158435 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:50.158568 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffeb67254264278ae7ba509bdb19e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeb67254264278ae7ba509bdb19e39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } } }
I20260924 19:36:50.158625 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:50.158732 5905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:50.159198 5905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:50.160925 5905 catalog_manager.cc:1347] Generated new cluster ID: f2906c2a5e45407c84f8404dce6d0eb5
I20260924 19:36:50.160979 5905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:50.163254 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:50.157381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.0.98.62:37767)
0924 19:36:50.157574 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:50.157583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:50.157615 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:36:50.157732 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:50.157787 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:50.157875 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:50.158251 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:50.158344 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.160104 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.161028 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.161613 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.161713 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:50.162538 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:50.162541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:50.162542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:36:50.162724 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:50.162762 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:36:50.162775 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:50.162818 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:50.162980 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:36:50.163052 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260924 19:36:50.164049 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:36:50.169543 5905 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:50.170225 5905 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:50.180847 5905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffeb67254264278ae7ba509bdb19e39: Generated new TSK 0
I20260924 19:36:50.181031 5905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:36:50.280349 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:50.187403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.0.98.62:37767)
0924 19:36:50.187456 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:50.187458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:50.187546 (+ 88us) server_negotiation.cc:407] Connection header received
0924 19:36:50.187598 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:50.187599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:50.187621 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:50.187698 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:36:50.187781 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.188369 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.189464 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.189653 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.189696 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:50.189856 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:36:50.280135 (+ 90279us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:36:50.280289 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58332: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 19:36:50.280411 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58332: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 19:36:50.280565 5920 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:50.187337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37767 (local address 127.0.0.1:58332)
0924 19:36:50.187484 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:50.187494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:50.187546 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:50.187694 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:50.187695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:50.187697 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:36:50.187744 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:50.187746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.188444 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.188445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:50.189390 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:50.189394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:50.189713 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:50.189719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:50.189777 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:50.189783 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:36:50.280317 (+ 90534us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:36:50.280469 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260924 19:36:50.280898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5824
2026-09-24T19:36:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (271 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:50 dist-test-slave-6kf3 krb5kdc[5932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:50 dist-test-slave-6kf3 krb5kdc[5932](info): set up 2 sockets
Sep 24 19:36:50 dist-test-slave-6kf3 krb5kdc[5932](info): commencing operation
krb5kdc: starting...
W20260924 19:36:51.395874 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:36:51.502440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:51.578862 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:51.579061 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:51.579082 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:51.580205 5945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:51.580243 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:51.580256 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:51.580268 5945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:51.580278 5945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:51.581703 5945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:51.581893 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:51.582060 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:51.586468 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.586478 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.586495 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:51.586493 5945 server_base.cc:1034] running on GCE node
I20260924 19:36:51.586920 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:51.587186 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:51.588331 5945 hybrid_clock.cc:648] HybridClock initialized: now 1790278611588313 us; error 32 us; skew 500 ppm
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:51.590662 5945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:36:51.591056 5945 webserver.cc:460] Webserver started at http://127.0.98.62:38309/ using document root <none> and password file <none>
I20260924 19:36:51.591241 5945 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:51.591286 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:51.591413 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:51.592514 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "5624abb1f770428ba6d325fe7f1b28c6"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "e164b05a5d88a86489784c410324b265"
server_key_iv: "0ec1f22e11e6c892d9cab03598f4f409"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.592831 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "5624abb1f770428ba6d325fe7f1b28c6"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "e164b05a5d88a86489784c410324b265"
server_key_iv: "0ec1f22e11e6c892d9cab03598f4f409"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.593977 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:51.594555 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.594714 5945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:51.594792 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "5624abb1f770428ba6d325fe7f1b28c6"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "e164b05a5d88a86489784c410324b265"
server_key_iv: "0ec1f22e11e6c892d9cab03598f4f409"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.594849 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:51.604498 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:51.606135 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:51.606263 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:51.609865 5945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43267
I20260924 19:36:51.609895 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43267 every 8 connection(s)
I20260924 19:36:51.610344 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:51.610929 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:51.613125 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6: Bootstrap starting.
I20260924 19:36:51.613657 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:51.613902 6013 log.cc:824] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:51.614491 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6: No bootstrap required, opened a new log
I20260924 19:36:51.616034 6013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260924 19:36:51.616166 6013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:51.616186 6013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5624abb1f770428ba6d325fe7f1b28c6, State: Initialized, Role: FOLLOWER
I20260924 19:36:51.616341 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260924 19:36:51.616436 6013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:51.616474 6013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:51.616503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 5945
I20260924 19:36:51.616520 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:51.616603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:51.616809 392 external_mini_cluster.cc:1428] Setting key cb4e9a7077a2824ea352666b290e984f
I20260924 19:36:51.617559 6013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260924 19:36:51.617645 6013 leader_election.cc:302] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5624abb1f770428ba6d325fe7f1b28c6; no voters:
I20260924 19:36:51.617784 6013 leader_election.cc:288] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:51.617821 6018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:51.617954 6018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5624abb1f770428ba6d325fe7f1b28c6, State: Running, Role: LEADER
I20260924 19:36:51.618081 6013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:51.618075 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } }
I20260924 19:36:51.618413 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5624abb1f770428ba6d325fe7f1b28c6. Latest consensus state: current_term: 1 leader_uuid: "5624abb1f770428ba6d325fe7f1b28c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260924 19:36:51.618620 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:51.618547 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5624abb1f770428ba6d325fe7f1b28c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5624abb1f770428ba6d325fe7f1b28c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43267 } } }
I20260924 19:36:51.618847 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:51.619179 6027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:36:51.619694 6027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:51.621979 6027 catalog_manager.cc:1347] Generated new cluster ID: 6d3a8af63c8e4bf8999bd40b3bb9df3b
I20260924 19:36:51.622031 6027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:51.627568 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:51.617253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.98.62:43267)
0924 19:36:51.617421 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:51.617429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:51.617457 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:36:51.617546 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:51.617551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:51.617591 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:51.617765 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:51.618510 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.619881 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.620763 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.621347 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.621449 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:51.624190 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:51.624194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:51.624194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:51.624206 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:51.626010 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.626414 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.626417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.626417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.626519 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.626717 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.626717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.626718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.626985 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:51.627123 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:51.627257 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:36:51.627383 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260924 19:36:51.630939 6027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:51.631644 6027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:51.639093 6027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5624abb1f770428ba6d325fe7f1b28c6: Generated new TSK 0
I20260924 19:36:51.639271 6027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:36:51.664984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:51.743242 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:51.743448 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:51.743479 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:51.744673 6039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:51.744726 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:51.744762 6039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:51.746234 6039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:51.746464 6039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:51.746666 6039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:51.751262 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.751308 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.751267 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:51.751386 6039 server_base.cc:1034] running on GCE node
I20260924 19:36:51.751722 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:51.751992 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:51.753146 6039 hybrid_clock.cc:648] HybridClock initialized: now 1790278611753127 us; error 34 us; skew 500 ppm
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:51.755452 6039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:36:51.755947 6039 webserver.cc:460] Webserver started at http://127.0.98.1:39279/ using document root <none> and password file <none>
I20260924 19:36:51.756160 6039 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:51.756206 6039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:51.756327 6039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:51.757321 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "2933ad7f2ddc4cf79de3d644804ff1e2"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "a1dd31f1910a6626ae37b9f487ee3896"
server_key_iv: "176f9bca6b08b5a938b1bbeb0736f7ef"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.757653 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2933ad7f2ddc4cf79de3d644804ff1e2"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "a1dd31f1910a6626ae37b9f487ee3896"
server_key_iv: "176f9bca6b08b5a938b1bbeb0736f7ef"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.758854 6039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:51.759595 6054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.759749 6039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:51.759835 6039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "2933ad7f2ddc4cf79de3d644804ff1e2"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "a1dd31f1910a6626ae37b9f487ee3896"
server_key_iv: "176f9bca6b08b5a938b1bbeb0736f7ef"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.759908 6039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:51.776175 6039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:51.777992 6039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:51.778143 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:51.778384 6039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:51.778714 6039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:51.778751 6039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.778793 6039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:51.778820 6039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.784793 6039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38615
I20260924 19:36:51.784832 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38615 every 8 connection(s)
I20260924 19:36:51.785142 6039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:36:51.789520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6039
I20260924 19:36:51.789618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:51.789826 392 external_mini_cluster.cc:1428] Setting key 8bf71bdbbb204c0c841d93deadc412bc
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:51.794361 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:51.785894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47553 (local address 127.0.98.62:43267)
0924 19:36:51.785964 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:51.785966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:51.786118 (+ 152us) server_negotiation.cc:407] Connection header received
0924 19:36:51.787147 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:51.787148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:51.787160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:51.787218 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:51.787838 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.788373 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.789346 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.789528 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.789570 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:51.791960 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:51.791961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:51.791961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:51.791970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:51.793132 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.793488 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.793489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.793490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.793543 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.793755 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.793756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.793756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.793867 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:51.793921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:51.794226 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:36:51.794298 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:36:51.794598 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:51.785833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.1:47553)
0924 19:36:51.785992 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:51.786033 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:51.787023 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:51.787278 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:51.787281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:51.787538 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:51.787741 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:51.787746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.788458 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.788459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:51.789267 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:51.789269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.789599 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.789600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:51.789753 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:51.790297 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:51.790305 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:51.791856 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:51.793211 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:51.793218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:51.793221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:51.793423 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:51.793595 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:51.793596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:51.793596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:51.793700 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:51.793920 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:51.793927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:51.794078 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 19:36:51.794188 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:36:51.794369 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 19:36:51.795065 6168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260924 19:36:51.795154 6168 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:51.795321 6168 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260924 19:36:51.795794 5977 ts_manager.cc:194] Registered new tserver with Master: 2933ad7f2ddc4cf79de3d644804ff1e2 (127.0.98.1:38615)
I20260924 19:36:51.796411 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47553
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:36:51.812186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:51.891260 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:51.891466 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:51.891487 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:51.892643 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:51.892680 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:51.892704 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:51.894114 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:51.894295 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:51.894470 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:51.899113 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.899144 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:51.899117 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:51.899400 6173 server_base.cc:1034] running on GCE node
I20260924 19:36:51.899587 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:51.899780 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:51.900929 6173 hybrid_clock.cc:648] HybridClock initialized: now 1790278611900911 us; error 31 us; skew 500 ppm
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:51.903352 6173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:36:51.903792 6173 webserver.cc:460] Webserver started at http://127.0.98.2:35261/ using document root <none> and password file <none>
I20260924 19:36:51.904036 6173 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:51.904096 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:51.904222 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:51.905233 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "725dd0f5c7c848738070c37ddbcfc1c3"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "c217deae4e842d5553186899c5c9b1b3"
server_key_iv: "975670f88849f3241bc25182876ef0ab"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.905560 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "725dd0f5c7c848738070c37ddbcfc1c3"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "c217deae4e842d5553186899c5c9b1b3"
server_key_iv: "975670f88849f3241bc25182876ef0ab"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.906801 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:36:51.907454 6188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.907608 6173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:51.907688 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "725dd0f5c7c848738070c37ddbcfc1c3"
format_stamp: "Formatted at 2026-09-24 19:36:51 on dist-test-slave-6kf3"
server_key: "c217deae4e842d5553186899c5c9b1b3"
server_key_iv: "975670f88849f3241bc25182876ef0ab"
server_key_version: "encryptionkey@0"
I20260924 19:36:51.907755 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:51.921011 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:51.922760 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:51.922891 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:51.923105 6173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:51.923393 6173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:51.923425 6173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.923452 6173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:51.923470 6173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:51.928681 6173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46043
I20260924 19:36:51.928743 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46043 every 8 connection(s)
I20260924 19:36:51.929018 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:36:51 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:51.936739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6173
I20260924 19:36:51.936842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:51.937012 392 external_mini_cluster.cc:1428] Setting key e83df48464ae077f793242b3efe39b99
I20260924 19:36:51.937227 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:51.929658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50639 (local address 127.0.98.62:43267)
0924 19:36:51.929703 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:51.929705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:51.929905 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:36:51.930788 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:51.930789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:51.930797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:51.930845 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:51.931388 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.931875 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.932868 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.933054 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.933105 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:51.934956 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:51.934957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:51.934957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:51.934964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:51.936087 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.936425 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.936425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.936426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.936468 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:51.936631 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:51.936631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:51.936632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:51.936759 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:51.936825 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:51.937075 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 19:36:51.937152 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:51.937398 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:51.929642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.2:50639)
0924 19:36:51.929792 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:51.929832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:51.930709 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:51.930906 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:51.930909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:51.931145 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:51.931326 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:51.931330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.931979 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.931980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:51.932790 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:51.932792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:51.933115 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:51.933115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:51.933211 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:51.933635 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:51.933643 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:51.934877 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:51.936150 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:51.936156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:51.936157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:51.936370 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:51.936502 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:51.936502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:51.936503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:51.936583 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:51.936819 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:51.936823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:51.936955 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:36:51.937035 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:36:51.937184 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260924 19:36:51.937918 6302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260924 19:36:51.937999 6302 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:51.938158 6302 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260924 19:36:51.938504 5977 ts_manager.cc:194] Registered new tserver with Master: 725dd0f5c7c848738070c37ddbcfc1c3 (127.0.98.2:46043)
I20260924 19:36:51.938933 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50639
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:36:51.959128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:52.037245 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:52.037451 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:52.037480 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:52.038637 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:52.038679 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:52.038712 6307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:52.040226 6307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:52.040455 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:52.040659 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:52.045291 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:52.045379 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:52.045285 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:52.045483 6307 server_base.cc:1034] running on GCE node
I20260924 19:36:52.045674 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:52.045903 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:52.047041 6307 hybrid_clock.cc:648] HybridClock initialized: now 1790278612047028 us; error 30 us; skew 500 ppm
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.049285 6307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:36:52.049675 6307 webserver.cc:460] Webserver started at http://127.0.98.3:43569/ using document root <none> and password file <none>
I20260924 19:36:52.049845 6307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:52.049882 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:52.049984 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:52.050985 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "bc9ced02dc55405caba846fffc8106b3"
format_stamp: "Formatted at 2026-09-24 19:36:52 on dist-test-slave-6kf3"
server_key: "ec824d3bedf1dd2c6f0ed8cf664b90a1"
server_key_iv: "331ad5e5880a276a7b798c1b5a1d89c0"
server_key_version: "encryptionkey@0"
I20260924 19:36:52.051270 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bc9ced02dc55405caba846fffc8106b3"
format_stamp: "Formatted at 2026-09-24 19:36:52 on dist-test-slave-6kf3"
server_key: "ec824d3bedf1dd2c6f0ed8cf664b90a1"
server_key_iv: "331ad5e5880a276a7b798c1b5a1d89c0"
server_key_version: "encryptionkey@0"
I20260924 19:36:52.052425 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:52.053005 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:52.053141 6307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:52.053231 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "bc9ced02dc55405caba846fffc8106b3"
format_stamp: "Formatted at 2026-09-24 19:36:52 on dist-test-slave-6kf3"
server_key: "ec824d3bedf1dd2c6f0ed8cf664b90a1"
server_key_iv: "331ad5e5880a276a7b798c1b5a1d89c0"
server_key_version: "encryptionkey@0"
I20260924 19:36:52.053285 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:52.061825 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:52.063498 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:52.063613 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:52.063795 6307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:52.064079 6307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:52.064111 6307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:52.064137 6307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:52.064162 6307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:52.070130 6307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41807
I20260924 19:36:52.070166 6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41807 every 8 connection(s)
I20260924 19:36:52.070454 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:52.072930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6307
I20260924 19:36:52.073016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:52.073194 392 external_mini_cluster.cc:1428] Setting key c6a86711c7dbf7064524f2e54c61ba8b
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:36:52.078769 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:52.071088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50951 (local address 127.0.98.62:43267)
0924 19:36:52.071134 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.071136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.071307 (+ 171us) server_negotiation.cc:407] Connection header received
0924 19:36:52.072222 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.072223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.072232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.072267 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.072843 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.073503 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.074533 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.074721 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.074771 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:52.076686 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:52.076687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:52.076688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:52.076696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:52.077787 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.078102 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.078102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.078102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.078153 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.078329 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.078330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.078330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.078419 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:52.078462 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:52.078653 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 19:36:52.078708 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:52.078920 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:52.071065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43267 (local address 127.0.98.3:50951)
0924 19:36:52.071199 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:52.071232 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:52.072116 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:52.072326 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:52.072329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:52.072553 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:52.072774 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:52.072778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.073572 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.073573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:52.074472 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:52.074475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.074779 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.074779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:52.074874 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:52.075341 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:52.075349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:52.076608 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:52.077847 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:52.077853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:52.077854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:52.078048 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:52.078208 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:52.078208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:52.078208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:52.078289 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:52.078458 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:52.078461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:52.078562 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 19:36:52.078634 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:36:52.078742 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260924 19:36:52.079335 6436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43267
I20260924 19:36:52.079419 6436 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:52.079577 6436 heartbeater.cc:511] Master 127.0.98.62:43267 requested a full tablet report, sending...
I20260924 19:36:52.079895 5977 ts_manager.cc:194] Registered new tserver with Master: bc9ced02dc55405caba846fffc8106b3 (127.0.98.3:41807)
I20260924 19:36:52.080251 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50951
I20260924 19:36:52.084779 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/test-admin.keytab.
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.099469 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:36:52.100971 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:52.100336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.0.98.1:38615)
0924 19:36:52.100480 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.100484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.100494 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:52.100520 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.100521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.100535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.100597 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.100890 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47716: BlockingRecv error: recv got EOF from 127.0.0.1:47716 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
W20260924 19:36:52.101075 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.101948 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.102715 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:52.110114 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:52.103313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.0.98.1:38615)
0924 19:36:52.103350 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.103352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.103362 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:52.103399 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.103400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.103408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.103453 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.103803 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.104652 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.105381 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.105720 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.105770 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:52.107621 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:52.107630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:52.107630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:52.107637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:52.109190 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.109478 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.109482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.109482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.109533 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.109659 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.109660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.109660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.109792 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:52.109898 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:52.109950 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:36:52.110012 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:36:52.111794 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:52.111351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.0.98.1:38615)
0924 19:36:52.111390 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.111391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.111402 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:36:52.111479 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.111480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.111489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.111530 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.111749 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47732: BlockingRecv error: recv got EOF from 127.0.0.1:47732 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:36:52.112018 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc)
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.112958 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.113734 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:52.120213 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:52.114244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.98.1:38615)
0924 19:36:52.114288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.114289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.114296 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:52.114320 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.114321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.114330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.114391 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.114718 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.115210 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.115827 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.116054 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.116099 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:52.118003 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:52.118005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:52.118005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:52.118012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:52.119294 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.119620 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.119621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.119622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.119670 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.119803 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.119804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.119804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.119885 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:52.119990 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:52.120079 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:36:52.120140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 19:36:52.121615 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:36:52.121104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.98.1:38615)
0924 19:36:52.121159 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.121161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.121169 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:52.121226 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.121227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.121235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.121278 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.121571 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47744: BlockingRecv error: recv got EOF from 127.0.0.1:47744 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 19:36:52.121667 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278584646480-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.122543 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:52.123271 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:36:52.129889 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:52.123775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.0.98.1:38615)
0924 19:36:52.123827 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:52.123829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:52.123837 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:52.123885 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:52.123886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:52.123893 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:52.123953 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:52.124353 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.125074 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.125731 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:52.125910 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:52.125953 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:52.127892 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:52.127893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:52.127894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:52.127900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:52.129001 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.129294 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.129295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.129295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.129342 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:52.129474 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:52.129475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:52.129475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:52.129595 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:52.129662 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:52.129746 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:36:52.129802 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:36:52.130525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6039
I20260924 19:36:52.134850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6173
I20260924 19:36:52.138537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6307
I20260924 19:36:52.141851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 5945
2026-09-24T19:36:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1693 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[6456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[6456](info): set up 2 sockets
Sep 24 19:36:52 dist-test-slave-6kf3 krb5kdc[6456](info): commencing operation
krb5kdc: starting...
W20260924 19:36:53.083796 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:36:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 19:36:53.188577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:53.266065 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:53.266258 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:53.266278 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:53.267392 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:53.267444 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:53.267458 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:53.267469 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:53.267480 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:53.268924 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:53.269155 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:53.269366 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.273995 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.274041 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.274194 6469 server_base.cc:1034] running on GCE node
W20260924 19:36:53.274048 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.274500 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.274767 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.275918 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790278613275902 us; error 27 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.282552 6469 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 19:36:53.282956 6469 webserver.cc:460] Webserver started at http://127.0.98.62:42679/ using document root <none> and password file <none>
I20260924 19:36:53.283134 6469 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.283172 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.283260 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:53.284286 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "0b95402c5e7d43098c928ceeeea67ea0"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "3dfbd267aa4d4bafc5b9594ee4fbc5e5"
server_key_iv: "504e7652ce8adc6a54b56f82588aab74"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.284574 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b95402c5e7d43098c928ceeeea67ea0"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "3dfbd267aa4d4bafc5b9594ee4fbc5e5"
server_key_iv: "504e7652ce8adc6a54b56f82588aab74"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.285702 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:53.286340 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.286470 6469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.286551 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "0b95402c5e7d43098c928ceeeea67ea0"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "3dfbd267aa4d4bafc5b9594ee4fbc5e5"
server_key_iv: "504e7652ce8adc6a54b56f82588aab74"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.286620 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.293754 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:53.295607 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:53.295717 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:53.299522 6469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33879
I20260924 19:36:53.299566 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33879 every 8 connection(s)
I20260924 19:36:53.299849 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:36:53.300865 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:53.302531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6469
I20260924 19:36:53.302616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:36:53.302809 392 external_mini_cluster.cc:1428] Setting key 17d1f84d80676185ef937364ced1efcf
I20260924 19:36:53.303009 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Bootstrap starting.
I20260924 19:36:53.303609 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:53.303854 6537 log.cc:824] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:53.304519 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: No bootstrap required, opened a new log
I20260924 19:36:53.306442 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } }
I20260924 19:36:53.306602 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:53.306639 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b95402c5e7d43098c928ceeeea67ea0, State: Initialized, Role: FOLLOWER
I20260924 19:36:53.306762 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } }
I20260924 19:36:53.306834 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:36:53.306864 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:36:53.306908 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:53.307462 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } }
I20260924 19:36:53.307557 6537 leader_election.cc:302] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b95402c5e7d43098c928ceeeea67ea0; no voters:
I20260924 19:36:53.307739 6537 leader_election.cc:288] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:36:53.307776 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:53.307920 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 0b95402c5e7d43098c928ceeeea67ea0, State: Running, Role: LEADER
I20260924 19:36:53.308020 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:53.308094 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } }
I20260924 19:36:53.308390 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b95402c5e7d43098c928ceeeea67ea0. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } }
I20260924 19:36:53.308597 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:53.308533 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } }
I20260924 19:36:53.308979 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:53.309146 6551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:36:53.309561 6551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:36:53.311292 6551 catalog_manager.cc:1347] Generated new cluster ID: e2ab2bc2c5394b9f92837f8cfd4b114b
I20260924 19:36:53.311345 6551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:36:53.312965 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.303230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.0.98.62:33879)
0924 19:36:53.303394 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.303404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.303439 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:36:53.303864 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.303871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.303917 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.304077 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.304763 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.305615 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.306664 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.307140 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.307215 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.309959 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.309963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.309964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.309976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.311641 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.311968 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.311971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.311972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.312059 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.312224 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.312224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.312225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.312454 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.312590 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.312676 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.312800 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260924 19:36:53.317534 6551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:36:53.318022 6551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:36:53.325143 6551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Generated new TSK 0
I20260924 19:36:53.325301 6551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 19:36:53.354970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:53.432123 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:53.432313 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:53.432334 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:53.433447 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:53.433486 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:53.433511 6563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:53.434933 6563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:53.435146 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:53.435352 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.439837 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.439855 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.439844 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.440094 6563 server_base.cc:1034] running on GCE node
I20260924 19:36:53.440284 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.440490 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.441629 6563 hybrid_clock.cc:648] HybridClock initialized: now 1790278613441623 us; error 29 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.443917 6563 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:36:53.444356 6563 webserver.cc:460] Webserver started at http://127.0.98.1:45235/ using document root <none> and password file <none>
I20260924 19:36:53.444552 6563 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.444598 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.444717 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:53.445699 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bbbed92bd404586a62f3dfe069860a3"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "120ceca457e407d8c951b1e3d729916d"
server_key_iv: "81e6c4c492caa96df886d53d1eec96b6"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.446036 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bbbed92bd404586a62f3dfe069860a3"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "120ceca457e407d8c951b1e3d729916d"
server_key_iv: "81e6c4c492caa96df886d53d1eec96b6"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.447184 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:36:53.447777 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.447918 6563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.448043 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "3bbbed92bd404586a62f3dfe069860a3"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "120ceca457e407d8c951b1e3d729916d"
server_key_iv: "81e6c4c492caa96df886d53d1eec96b6"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.448107 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.461649 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:53.463241 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:53.463366 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:53.463557 6563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:53.463837 6563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:53.463874 6563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.463901 6563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:53.463919 6563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.469146 6563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42241
I20260924 19:36:53.469206 6691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42241 every 8 connection(s)
I20260924 19:36:53.469461 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:36:53.477626 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.470079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34835 (local address 127.0.98.62:33879)
0924 19:36:53.470129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.470131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.470291 (+ 160us) server_negotiation.cc:407] Connection header received
0924 19:36:53.470791 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.470792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.470804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.470849 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.471723 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.472269 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.473202 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.473397 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.473449 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.475292 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.475294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.475294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.475301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.476465 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.476799 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.476800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.476801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.476849 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.477055 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.477056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.477056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.477154 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.477207 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.477449 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.477539 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:36:53.477771 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.470045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:34835)
0924 19:36:53.470181 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.470219 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.470701 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.470930 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.470934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.471452 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.471635 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.471639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.472346 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.472347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.473123 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.473125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.473461 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.473462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.473583 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.474005 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.474013 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.475212 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.476532 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.476538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.476548 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.476739 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.476901 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.476901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.476902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.476997 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.477205 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.477211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.477337 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.477423 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.477571 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260924 19:36:53.478223 6692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:53.478298 6692 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:53.478456 6692 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:53.478909 6501 ts_manager.cc:194] Registered new tserver with Master: 3bbbed92bd404586a62f3dfe069860a3 (127.0.98.1:42241)
I20260924 19:36:53.479159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6563
I20260924 19:36:53.479247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:36:53.479409 392 external_mini_cluster.cc:1428] Setting key 3826c68e7dce2df2e37b9bc9fd03bb47
I20260924 19:36:53.479610 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34835
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 19:36:53.500625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:53.578655 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:53.578917 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:53.578953 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:53.580605 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:53.580663 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:53.580703 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:53.582763 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:53.583009 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:53.583249 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.588267 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.588274 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.588320 6697 server_base.cc:1034] running on GCE node
W20260924 19:36:53.588307 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.588629 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.588846 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.589984 6697 hybrid_clock.cc:648] HybridClock initialized: now 1790278613589973 us; error 25 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.592274 6697 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:36:53.592753 6697 webserver.cc:460] Webserver started at http://127.0.98.2:37017/ using document root <none> and password file <none>
I20260924 19:36:53.592971 6697 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.593024 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.593152 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:53.594266 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "31af10e1de594a5c91e190d7b9d2f1c1"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "4f5af51ac435eeb9b705823e82d16045"
server_key_iv: "b4b1832b5479b947523205a1377ddd7e"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.594615 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31af10e1de594a5c91e190d7b9d2f1c1"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "4f5af51ac435eeb9b705823e82d16045"
server_key_iv: "b4b1832b5479b947523205a1377ddd7e"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.595870 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:36:53.596592 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.596755 6697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:53.596836 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "31af10e1de594a5c91e190d7b9d2f1c1"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "4f5af51ac435eeb9b705823e82d16045"
server_key_iv: "b4b1832b5479b947523205a1377ddd7e"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.596894 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.612671 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:53.614270 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:53.614394 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:53.614596 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:53.614871 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:53.614907 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.614928 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:53.614950 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.620442 6697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37301
I20260924 19:36:53.620491 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37301 every 8 connection(s)
I20260924 19:36:53.620781 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:53.624552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6697
I20260924 19:36:53.624644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:36:53.624835 392 external_mini_cluster.cc:1428] Setting key 6570df30ee1fc4939d2fa814a8fb4a6f
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:36:53.629319 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.621377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38111 (local address 127.0.98.62:33879)
0924 19:36:53.621430 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.621432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.621614 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:36:53.622062 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.622063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.622072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.622114 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.622941 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.623416 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.624371 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.624555 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.624605 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.626831 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.626832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.626833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.626840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.628107 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.628499 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.628500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.628501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.628554 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.628754 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.628755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.628756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.628862 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.628915 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.629157 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.629234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:36:53.629482 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.621355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:38111)
0924 19:36:53.621498 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.621538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.621985 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.622174 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.622177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.622699 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.622880 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.622884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.623471 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.623471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.624312 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.624314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.624629 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.624630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.624772 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.625251 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.625259 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.626746 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.628180 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.628186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.628188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.628436 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.628604 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.628605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.628605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.628704 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.628912 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.628918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.629047 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.629131 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.629280 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260924 19:36:53.630025 6826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:53.630100 6826 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:53.630250 6826 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:53.630613 6500 ts_manager.cc:194] Registered new tserver with Master: 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
I20260924 19:36:53.631105 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38111
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:36:53.645720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:53.723301 6831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:53.723492 6831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:53.723513 6831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:53.724612 6831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:53.724647 6831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:53.724673 6831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:53.726032 6831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:53.726205 6831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:53.726373 6831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.731101 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.731115 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.731101 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.731287 6831 server_base.cc:1034] running on GCE node
I20260924 19:36:53.731482 6831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.731722 6831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.732853 6831 hybrid_clock.cc:648] HybridClock initialized: now 1790278613732840 us; error 24 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.735095 6831 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:36:53.735469 6831 webserver.cc:460] Webserver started at http://127.0.98.3:36475/ using document root <none> and password file <none>
I20260924 19:36:53.735671 6831 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.735710 6831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.735798 6831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:53.736814 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "551edc6f070f45b8827773515d61c575"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "759a8948f8d21d73365560a4be98d06d"
server_key_iv: "ae4d26f405764c975265c5da65e23fde"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.737090 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "551edc6f070f45b8827773515d61c575"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "759a8948f8d21d73365560a4be98d06d"
server_key_iv: "ae4d26f405764c975265c5da65e23fde"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.738234 6831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:53.738854 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.738996 6831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:53.739069 6831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "551edc6f070f45b8827773515d61c575"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "759a8948f8d21d73365560a4be98d06d"
server_key_iv: "ae4d26f405764c975265c5da65e23fde"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.739137 6831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.747484 6831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:53.749166 6831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:53.749271 6831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:53.749477 6831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:53.749734 6831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:53.749763 6831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.749782 6831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:53.749799 6831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.754926 6831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36729
I20260924 19:36:53.754987 6959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36729 every 8 connection(s)
I20260924 19:36:53.755244 6831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:53.759598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6831
I20260924 19:36:53.759683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:36:53.759842 392 external_mini_cluster.cc:1428] Setting key 5fb0a362d2f837591c7f4a8e94b2fa47
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:36:53.763049 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.755864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43139 (local address 127.0.98.62:33879)
0924 19:36:53.755964 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.755966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.756124 (+ 158us) server_negotiation.cc:407] Connection header received
0924 19:36:53.756571 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.756572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.756581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.756625 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.757456 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.757933 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.758844 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.759030 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.759079 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.761062 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.761063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.761063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.761070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.762086 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.762395 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.762395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.762396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.762437 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.762605 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.762605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.762606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.762697 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.762739 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.762928 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.762981 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:36:53.763275 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.755839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:43139)
0924 19:36:53.756014 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.756050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.756488 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.756684 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.756687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.757215 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.757395 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.757399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.757989 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.757990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.758787 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.758790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.759086 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.759086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.759183 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.759632 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.759642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.760982 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.762145 (+ 1163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.762150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.762151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.762343 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.762478 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.762478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.762479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.762566 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.762737 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.762739 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.762839 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.762910 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.763019 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260924 19:36:53.763690 6960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:53.763767 6960 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:53.763900 6960 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:53.764329 6500 ts_manager.cc:194] Registered new tserver with Master: 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729)
I20260924 19:36:53.764711 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43139
I20260924 19:36:53.771311 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 19:36:53.792457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41695
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33879,127.0.98.61:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:53.869650 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:53.869845 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:53.869874 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:53.870987 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:53.871023 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:53.871042 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:53.871057 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:53.871073 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:53.872575 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33879,127.0.98.61:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
W20260924 19:36:53.872684 6965 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33879,127.0.98.61:41695'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:36:53.872825 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:53.873008 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.877300 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.877318 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.877347 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.877473 6965 server_base.cc:1034] running on GCE node
I20260924 19:36:53.877665 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.877902 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.879031 6965 hybrid_clock.cc:648] HybridClock initialized: now 1790278613879019 us; error 26 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.881285 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 19:36:53.881647 6965 webserver.cc:460] Webserver started at http://127.0.98.61:37957/ using document root <none> and password file <none>
I20260924 19:36:53.881814 6965 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.881850 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.881933 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:53.882858 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/instance:
uuid: "681be2f83c664d9788066b058eae4063"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "48a69759d2f5f2acc7123f7d2a018ec5"
server_key_iv: "647fbb0bbedc88691dd77ba7d64156a7"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.883138 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal/instance:
uuid: "681be2f83c664d9788066b058eae4063"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "48a69759d2f5f2acc7123f7d2a018ec5"
server_key_iv: "647fbb0bbedc88691dd77ba7d64156a7"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.884268 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:53.884855 6980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.884994 6965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:53.885073 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "681be2f83c664d9788066b058eae4063"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "48a69759d2f5f2acc7123f7d2a018ec5"
server_key_iv: "647fbb0bbedc88691dd77ba7d64156a7"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.885135 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.901615 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:53.903599 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:53.903721 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:53.907176 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41695
I20260924 19:36:53.907210 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41695 every 8 connection(s)
I20260924 19:36:53.907501 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:36:53.908427 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:53.909873 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:36:53.916254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 6965
I20260924 19:36:53.917776 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.910519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.98.62:33879)
0924 19:36:53.910579 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.910581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.910758 (+ 177us) server_negotiation.cc:407] Connection header received
0924 19:36:53.911179 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.911180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.911190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.911237 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.911785 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.912298 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.913210 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.913381 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.913420 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.915246 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.915247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.915247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.915255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.916304 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.916841 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.916843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.916843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.916904 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.917132 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.917133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.917134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.917327 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.917397 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.917652 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.917716 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:36:53.917976 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.910503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49822)
0924 19:36:53.910651 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.910683 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.911104 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.911297 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.911300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.911544 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.911724 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.911727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.912360 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.912361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.913150 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.913152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.913429 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.913429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.913535 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.913892 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.913899 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.915161 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.916432 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.916438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.916440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.916743 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.916966 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.916967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.916967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.917078 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.917398 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.917401 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.917520 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.917628 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.917772 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260924 19:36:53.918540 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:36:53.924310 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.916673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.0.98.61:41695)
0924 19:36:53.916823 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.916828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.916846 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:36:53.916864 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.916865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.916875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.916927 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.917525 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.918390 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.919217 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.919518 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.919563 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.921890 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.921897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.921898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.921904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.923178 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.923539 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.923541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.923541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.923595 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.923767 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.923768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.923768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.923891 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.924023 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.924110 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.924212 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260924 19:36:53.925189 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.918739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34644)
0924 19:36:53.918784 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.918792 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.918823 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.919017 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.919017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.919141 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.919177 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.919178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.919842 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.919843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.920377 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.920379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.920815 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.920816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.920861 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.921148 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.921153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.922310 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.924350 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.924351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.924352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.924502 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.924662 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.924662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.924662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.924718 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.924992 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.924992 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.925023 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.925070 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.925135 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:36:53.925293 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.918801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.0.98.61:41695)
0924 19:36:53.918920 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.918921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.918926 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:36:53.918939 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.918939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.918946 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.918995 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.919244 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.919776 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.920446 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.920750 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.920818 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.922390 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.922392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.922392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.922397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.924279 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.924565 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.924566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.924566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.924611 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.924840 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.924841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.924841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.924936 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.924983 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.925085 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.925178 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260924 19:36:53.926342 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Bootstrap starting.
I20260924 19:36:53.926867 7033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:53.927114 7033 log.cc:824] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:53.927716 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: No bootstrap required, opened a new log
I20260924 19:36:53.929049 7033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } }
I20260924 19:36:53.929158 7033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:53.929178 7033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681be2f83c664d9788066b058eae4063, State: Initialized, Role: FOLLOWER
I20260924 19:36:53.929276 7033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } }
I20260924 19:36:53.929495 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } } }
I20260924 19:36:53.929544 7033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:53.929580 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:36:53.930925 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:53.930974 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:53.932371 6965 master_runner.cc:428] Detected that this master 681be2f83c664d9788066b058eae4063 is joining an existing cluster
I20260924 19:36:53.932417 6965 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41695
I20260924 19:36:53.938616 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.933300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34654)
0924 19:36:53.933455 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.933464 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.933508 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.933607 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.933607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.933748 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.933812 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.933814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.934392 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.934393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.934900 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.934902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.935212 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.935213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.935273 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.935631 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.935636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.936015 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.937648 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.937649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.937649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.937848 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.938022 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.938023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.938072 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.938312 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.938312 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.938344 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.938405 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.938505 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260924 19:36:53.938670 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.933470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.0.98.61:41695)
0924 19:36:53.933523 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.933525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.933531 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:36:53.933548 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.933549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.933559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.933605 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.933861 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.934318 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.934963 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.935158 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.935207 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.936079 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.936081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.936081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.936087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.937567 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.937918 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.937919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.937919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.937963 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938130 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.938130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.938131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.938235 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.938272 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.938440 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.938556 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":40}
I20260924 19:36:53.939260 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.933328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.0.98.62:33879)
0924 19:36:53.933366 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.933368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.933631 (+ 263us) server_negotiation.cc:407] Connection header received
0924 19:36:53.933715 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.933716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.933726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.933776 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.934089 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.934828 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.935841 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.936144 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.936196 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.937020 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.937021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.937022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.937027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.938178 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938477 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.938478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.938478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.938530 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938710 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.938711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.938712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.938832 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.938869 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.939094 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.939171 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:53.939495 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.933299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49828)
0924 19:36:53.933440 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.933451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.933649 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.933780 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.933781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.933948 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.934001 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.934003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.934917 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.934918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.935767 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.935769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.936199 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.936200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.936271 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.936628 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.936632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.936940 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.938247 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.938248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.938398 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.938594 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.938595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.938595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.938648 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.938912 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.938913 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.938954 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.939032 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.939394 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:36:53.947639 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.941265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.0.98.61:41695)
0924 19:36:53.941295 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.941297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.941445 (+ 148us) server_negotiation.cc:407] Connection header received
0924 19:36:53.941518 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.941519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.941527 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.941569 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.941951 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.942405 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.943161 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.943361 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.943410 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.945420 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.945421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.945421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.945428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.946660 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.946984 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.946985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.946985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.947033 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.947182 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.947182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.947183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.947275 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.947324 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.947477 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.947557 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:36:53.947662 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.941224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34660)
0924 19:36:53.941363 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.941392 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.941454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.941608 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.941612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.941842 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.941885 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.941886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.942469 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.942470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.943096 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.943098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.943417 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.943417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.943493 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.943869 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.943872 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.945301 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.946738 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.946745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.946747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.946922 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.947084 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.947084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.947085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.947135 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.947325 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.947332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.947381 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.947448 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.947560 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 19:36:53.948242 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "681be2f83c664d9788066b058eae4063"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41695
}
attrs {
promote: true
}
}
dest_uuid: "0b95402c5e7d43098c928ceeeea67ea0"
cas_config_opid_index: -1
I20260924 19:36:53.948516 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } }
I20260924 19:36:53.948926 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 681be2f83c664d9788066b058eae4063 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } }
I20260924 19:36:53.949193 7007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:53.949685 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41695
I20260924 19:36:53.949862 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } }
I20260924 19:36:53.949929 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:53.950096 7007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Refusing update from remote peer 0b95402c5e7d43098c928ceeeea67ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:36:53.950234 6965 master.cc:537] Master@127.0.98.61:41695 shutting down...
I20260924 19:36:53.950639 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:36:53.951613 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 19:36:53.951965 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 -> Peer 681be2f83c664d9788066b058eae4063 (127.0.98.61:41695): Couldn't send request to peer 681be2f83c664d9788066b058eae4063. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:36:53.952754 6965 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:53.952836 6965 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:53.952879 6965 tablet_replica.cc:330] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: stopping tablet replica
I20260924 19:36:53.954833 6965 master.cc:552] Master@127.0.98.61:41695 shutdown complete.
I20260924 19:36:53.954867 6965 master_runner.cc:305] Clearing existing system tablet
I20260924 19:36:53.954984 6965 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.955022 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.955516 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:36:53.955922 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.956089 6965 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:53.956135 6965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.956158 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "681be2f83c664d9788066b058eae4063"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "48a69759d2f5f2acc7123f7d2a018ec5"
server_key_iv: "647fbb0bbedc88691dd77ba7d64156a7"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.956667 6965 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:53.957929 6965 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:36:53.957983 6965 log.cc:1196] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:36:53.958129 6965 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Deleting consensus metadata
I20260924 19:36:53.958226 6965 master_runner.cc:315] Copying system tablet from 127.0.98.62:33879
I20260924 19:36:53.958750 6965 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33879
I20260924 19:36:53.964727 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.959096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.0.98.62:33879)
0924 19:36:53.959138 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:53.959139 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:53.959202 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:36:53.959267 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:53.959268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:53.959275 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:53.959316 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:53.959613 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.960211 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.961203 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.961373 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.961412 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.962294 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:53.962296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:53.962296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:53.962303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:53.963686 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.964076 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.964076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.964077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.964127 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:53.964298 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:53.964298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:53.964299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:53.964420 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:53.964471 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:53.964610 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:36:53.964667 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:53.964788 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:53.959025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49842)
0924 19:36:53.959146 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:53.959157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:53.959215 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:53.959322 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:53.959323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:53.959502 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:53.959558 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.959560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.960274 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.960275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.961137 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:53.961139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:53.961424 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:53.961424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:53.961482 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:53.961840 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:53.961844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:53.962197 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:53.963762 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.963763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.963763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.964002 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.964185 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:53.964186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:53.964186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:53.964245 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:53.964474 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:53.964475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:53.964525 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:36:53.964592 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:36:53.964685 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 19:36:53.965037 6521 tablet_copy_service.cc:140] P 0b95402c5e7d43098c928ceeeea67ea0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 681be2f83c664d9788066b058eae4063 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49842)
I20260924 19:36:53.965130 6521 tablet_copy_service.cc:161] P 0b95402c5e7d43098c928ceeeea67ea0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 681be2f83c664d9788066b058eae4063 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49842: session id = 681be2f83c664d9788066b058eae4063-00000000000000000000000000000000
I20260924 19:36:53.965724 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:36:53.966285 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:36:53.967278 6965 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: tablet copy: Starting download of 0 data blocks...
I20260924 19:36:53.967417 6965 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: tablet copy: Starting download of 1 WAL segments...
I20260924 19:36:53.968209 6965 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:36:53.968945 6521 tablet_copy_service.cc:342] P 0b95402c5e7d43098c928ceeeea67ea0: Request end of tablet copy session 681be2f83c664d9788066b058eae4063-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49842
I20260924 19:36:53.968998 6521 tablet_copy_service.cc:433] P 0b95402c5e7d43098c928ceeeea67ea0: ending tablet copy session 681be2f83c664d9788066b058eae4063-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 681be2f83c664d9788066b058eae4063
I20260924 19:36:53.971431 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:53.972563 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.972718 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:53.972747 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:53.972949 6965 server_base.cc:1034] running on GCE node
I20260924 19:36:53.973049 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:53.973187 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:53.974301 6965 hybrid_clock.cc:648] HybridClock initialized: now 1790278613974292 us; error 22 us; skew 500 ppm
Sep 24 19:36:53 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:53.975627 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 19:36:53.975858 6965 webserver.cc:460] Webserver started at http://127.0.98.61:33781/ using document root <none> and password file <none>
I20260924 19:36:53.976054 6965 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:53.976084 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:53.976594 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.977049 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.977171 6965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:53.977226 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "681be2f83c664d9788066b058eae4063"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "48a69759d2f5f2acc7123f7d2a018ec5"
server_key_iv: "647fbb0bbedc88691dd77ba7d64156a7"
server_key_version: "encryptionkey@0"
I20260924 19:36:53.977277 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.020315 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.021843 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.024950 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41695
I20260924 19:36:54.024991 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41695 every 8 connection(s)
I20260924 19:36:54.025277 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:36:54.026113 7135 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:36:54.026505 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Bootstrap starting.
I20260924 19:36:54.028960 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:36:54.029139 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: Bootstrap complete.
I20260924 19:36:54.029321 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } }
I20260924 19:36:54.029393 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 681be2f83c664d9788066b058eae4063, State: Initialized, Role: LEARNER
I20260924 19:36:54.029429 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } }
I20260924 19:36:54.029598 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:54.029598 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } }
I20260924 19:36:54.029788 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:36:54.030975 7150 catalog_manager.cc:1259] Loaded cluster ID: e2ab2bc2c5394b9f92837f8cfd4b114b
I20260924 19:36:54.031116 7150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: loading cluster ID for follower catalog manager: success
I20260924 19:36:54.032220 7150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: acquiring CA information for follower catalog manager: success
I20260924 19:36:54.032545 7150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:36:54.033070 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.025587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.0.98.61:41695)
0924 19:36:54.025742 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.025744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.025752 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.025772 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.025773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.025781 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.025845 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.026506 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.027412 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.028434 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.028709 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.028783 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.030163 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.030165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.030165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.030171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.032000 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.032358 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.032359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.032360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.032423 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.032581 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.032582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.032583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.032697 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.032784 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.032892 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.032972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 19:36:54.058699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42337
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:36:54.147001 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:54.147202 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:54.147221 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:54.148377 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:36:54.148417 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:54.148430 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:54.148442 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:36:54.148464 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:36:54.149930 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:54.150151 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:54.150359 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:54.154773 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.154831 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.154775 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.155049 7153 server_base.cc:1034] running on GCE node
I20260924 19:36:54.155278 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:54.155561 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:54.156694 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790278614156677 us; error 27 us; skew 500 ppm
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:54.158937 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 19:36:54.159365 7153 webserver.cc:460] Webserver started at http://127.0.98.60:44697/ using document root <none> and password file <none>
I20260924 19:36:54.159579 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.159626 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.159742 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:36:54.160784 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/instance:
uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
format_stamp: "Formatted at 2026-09-24 19:36:54 on dist-test-slave-6kf3"
server_key: "392f58b14544e4c66be727941f2fa493"
server_key_iv: "05862e4f579cac195cae112a6e0e2dda"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.161105 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal/instance:
uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
format_stamp: "Formatted at 2026-09-24 19:36:54 on dist-test-slave-6kf3"
server_key: "392f58b14544e4c66be727941f2fa493"
server_key_iv: "05862e4f579cac195cae112a6e0e2dda"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.162266 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:36:54.162894 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.163051 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:54.163134 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
format_stamp: "Formatted at 2026-09-24 19:36:54 on dist-test-slave-6kf3"
server_key: "392f58b14544e4c66be727941f2fa493"
server_key_iv: "05862e4f579cac195cae112a6e0e2dda"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.163198 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.182214 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.184193 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:54.184329 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.187867 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42337
I20260924 19:36:54.187901 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42337 every 8 connection(s)
I20260924 19:36:54.188251 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:36:54.189208 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:54.190717 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:36:54.193528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 7153
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.199820 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.191485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.0.98.62:33879)
0924 19:36:54.191544 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.191546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.191610 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:36:54.192150 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.192151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.192162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.192224 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.192852 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.193348 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.194742 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.194987 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.195056 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.197283 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.197285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.197285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.197292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.198489 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.198877 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.198878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.198879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.198940 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.199195 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.199197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.199197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.199308 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.199367 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.199643 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.199734 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:36:54.200004 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.191336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49850)
0924 19:36:54.191494 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.191544 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.192056 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.192309 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.192312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.192578 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.192769 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.192773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.193532 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.193533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.194666 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.194669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.195063 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.195064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.195163 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.195596 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.195604 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.197215 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.198560 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.198566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.198568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.198813 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.199000 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.199000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.199000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.199102 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.199365 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.199371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.199521 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.199608 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.199774 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 19:36:54.200522 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:36:54.202064 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.193960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.0.98.60:42337)
0924 19:36:54.194097 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.194101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.194115 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:36:54.194131 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.194132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.194147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.194205 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.194838 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.195584 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.196485 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.196933 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.196997 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.199156 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.199162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.199163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.199169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.201098 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.201379 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.201382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.201382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.201433 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.201593 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.201594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.201594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.201720 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.201872 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.201911 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.201975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:36:54.205943 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.200732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34682)
0924 19:36:54.200770 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.200779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.200810 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.200940 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.200941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.201099 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.201125 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.201127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.201738 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.201739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.202303 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.202305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.202646 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.202647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.202690 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.202952 (+ 262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.202955 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.203892 (+ 937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.205254 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.205255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.205255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.205408 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.205559 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.205559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.205560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.205599 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.205751 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.205751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.205781 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.205823 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.205882 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:36:54.205948 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.200794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.61:41695)
0924 19:36:54.200837 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.200839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.200847 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.200869 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.200870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.200879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.200939 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.201196 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.201669 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.202362 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.202592 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.202649 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.204001 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.204003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.204003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.204011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.205193 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.205475 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.205476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.205476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.205517 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.205631 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.205631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.205631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.205714 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.205752 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.205838 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.205891 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:54.206250 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42337 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.211279 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.206435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45248)
0924 19:36:54.206465 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.206476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.206515 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.206603 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.206604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.206696 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.206720 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.206721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.207213 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.207213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.207659 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.207660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.207990 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.207990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.208025 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.208281 (+ 256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.208285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.209301 (+ 1016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.210618 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.210619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.210619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.210753 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.210880 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.210880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.210881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.210925 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.211070 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.211071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.211098 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.211140 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.211201 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 19:36:54.211331 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.206486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.98.60:42337)
0924 19:36:54.206517 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.206518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.206526 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.206558 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.206559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.206566 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.206599 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.206754 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.207168 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.207715 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.207904 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.207979 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.209374 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.209375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.209376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.209381 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.210560 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.210803 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.210803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.210804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.210841 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.210956 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.210956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.210956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.211033 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.211073 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.211156 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.211238 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":16}
I20260924 19:36:54.212996 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Bootstrap starting.
I20260924 19:36:54.213761 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:54.213994 7221 log.cc:824] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:54.214666 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: No bootstrap required, opened a new log
I20260924 19:36:54.216028 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42337 } }
I20260924 19:36:54.216156 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:54.216177 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d9cc31fcb4f59a3e4797b9aba9fc5, State: Initialized, Role: FOLLOWER
I20260924 19:36:54.216297 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42337 } }
I20260924 19:36:54.216569 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:54.216553 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42337 } } }
I20260924 19:36:54.216636 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:36:54.217970 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:36:54.218022 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:36:54.220012 7153 master_runner.cc:428] Detected that this master 441d9cc31fcb4f59a3e4797b9aba9fc5 is joining an existing cluster
I20260924 19:36:54.220050 7153 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42337
I20260924 19:36:54.226552 7226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:54.226656 7226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42337 } }
I20260924 19:36:54.226898 7226 leader_election.cc:288] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879), 681be2f83c664d9788066b058eae4063 (127.0.98.61:41695)
I20260924 19:36:54.227097 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681be2f83c664d9788066b058eae4063" is_pre_election: true
I20260924 19:36:54.227085 6511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b95402c5e7d43098c928ceeeea67ea0" is_pre_election: true
I20260924 19:36:54.227187 7109 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Handling vote request from an unknown peer 441d9cc31fcb4f59a3e4797b9aba9fc5
I20260924 19:36:54.227226 7109 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 441d9cc31fcb4f59a3e4797b9aba9fc5 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 19:36:54.227461 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.221101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.0.98.60:42337)
0924 19:36:54.221132 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.221134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.221240 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:36:54.221297 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.221297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.221307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.221359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.221825 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222526 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.223537 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.223811 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.223843 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.225134 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.225135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.225136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.225141 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.226346 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.226765 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.226766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.226767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.226814 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.226987 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.226988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.226988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.227101 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.227140 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.227306 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.227375 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:36:54.227487 7171 leader_election.cc:302] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 441d9cc31fcb4f59a3e4797b9aba9fc5; no voters: 0b95402c5e7d43098c928ceeeea67ea0, 681be2f83c664d9788066b058eae4063
I20260924 19:36:54.227690 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.220951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45252)
0924 19:36:54.221181 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.221190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.221249 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.221402 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.221403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.221678 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.221751 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.221753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222607 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.222608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.223454 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.223455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.223917 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.223917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.224014 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.224615 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.224620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.225052 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.226416 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.226417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.226418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.226688 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.226871 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.226872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.226872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.226918 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.227175 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.227176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.227217 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.227521 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.227603 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260924 19:36:54.227710 7226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:54.228515 7226 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:36:54.247691 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.221148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.61:41695)
0924 19:36:54.221183 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.221185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.221334 (+ 149us) server_negotiation.cc:407] Connection header received
0924 19:36:54.221379 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.221380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.221390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.221435 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.221889 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222628 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.223746 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.224060 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.224098 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.245242 (+ 21144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.245243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.245243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.245251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.246660 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.246954 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.246955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.246955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.247008 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247195 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.247195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.247196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.247329 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.247383 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.247519 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.247593 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:36:54.247716 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.220989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34690)
0924 19:36:54.221288 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.221296 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.221338 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.221473 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.221474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.221748 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.221805 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.221806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222725 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.222726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.223655 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.223657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.224170 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.224170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.224237 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.244543 (+ 20306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.244550 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.245102 (+ 552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.246738 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.246740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.246740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.246894 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.247070 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.247072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.247139 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.247381 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.247382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.247435 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.247491 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.247595 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":42,"thread_start_us":79,"threads_started":1}
I20260924 19:36:54.248078 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.221129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.98.62:33879)
0924 19:36:54.221167 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.221169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.221254 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:36:54.221475 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.221476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.221492 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.221518 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.221915 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222637 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.223746 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.224060 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.224098 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.245236 (+ 21138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.245238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.245239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.245248 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.246940 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247289 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.247290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.247290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.247345 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247532 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.247533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.247533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.247655 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.247693 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.247871 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.247986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:54.248368 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.220951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49852)
0924 19:36:54.221207 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.221215 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.221398 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.221577 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.221578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.221794 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.221839 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.221840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.222718 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.222718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.223655 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.223657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.224169 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.224170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.224237 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.244613 (+ 20376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.244617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.245102 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.247027 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.247028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.247178 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.247414 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.247415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.247415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.247467 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.247733 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.247734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.247772 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.248158 (+ 386us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.248262 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":42,"spinlock_wait_cycles":8576,"thread_start_us":100,"threads_started":1}
I20260924 19:36:54.253543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6563
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.256817 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.250202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45268)
0924 19:36:54.250256 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.250265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.250298 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.250405 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.250406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.250583 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.250631 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.250633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.251222 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.251223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.252084 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.252086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.252380 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.252381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.252446 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.252808 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.252813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.254251 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.255857 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.255858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.255859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.256079 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.256253 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.256253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.256254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.256311 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.256533 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.256534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.256575 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.256635 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.256723 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:36:54.256819 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.250257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.0.98.60:42337)
0924 19:36:54.250291 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.250293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.250301 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.250352 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.250353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.250360 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.250402 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.250693 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.251159 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.252152 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.252322 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.252372 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.254331 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.254332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.254333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.254340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.255723 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.256139 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.256140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.256141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.256194 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.256365 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.256366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.256366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.256480 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.256538 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.256663 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.256731 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:36:54.257427 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42337: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42337
}
attrs {
promote: true
}
}
dest_uuid: "0b95402c5e7d43098c928ceeeea67ea0"
cas_config_opid_index: 5
I20260924 19:36:54.257663 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:54.257985 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441d9cc31fcb4f59a3e4797b9aba9fc5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.258882 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42337
I20260924 19:36:54.258949 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.259018 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:54.259322 7153 master.cc:537] Master@127.0.98.60:42337 shutting down...
I20260924 19:36:54.259351 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 FOLLOWER]: Refusing update from remote peer 0b95402c5e7d43098c928ceeeea67ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:36:54.259586 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:36:54.260278 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 -> Peer 441d9cc31fcb4f59a3e4797b9aba9fc5 (127.0.98.60:42337): Couldn't send request to peer 441d9cc31fcb4f59a3e4797b9aba9fc5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:36:54.260840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33879
--webserver_port=45235
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:36:54.265725 7153 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:54.265810 7153 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:54.265851 7153 tablet_replica.cc:330] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: stopping tablet replica
I20260924 19:36:54.266227 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.259248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.98.61:41695)
0924 19:36:54.260429 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.260431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.260440 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:54.260461 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.260462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.260483 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.260543 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.260755 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.261411 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.262893 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.263354 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.263416 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.263501 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.266109 (+ 2608us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.266159 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156}
I20260924 19:36:54.266571 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.259150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.0.1:34692)
0924 19:36:54.259181 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.259192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.259228 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.260636 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.260637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.260640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.260677 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.260678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.262066 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.262067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.262809 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.262811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.265486 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.265487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.265566 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.265596 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.266467 (+ 871us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.266514 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 19:36:54.267146 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Refusing update from remote peer 0b95402c5e7d43098c928ceeeea67ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:36:54.267467 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 19:36:54.268157 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441d9cc31fcb4f59a3e4797b9aba9fc5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.268550 7153 master.cc:552] Master@127.0.98.60:42337 shutdown complete.
I20260924 19:36:54.268584 7153 master_runner.cc:305] Clearing existing system tablet
I20260924 19:36:54.268702 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.268736 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.269260 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b95402c5e7d43098c928ceeeea67ea0. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.269335 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:36:54.269366 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: true } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.269415 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:36:54.269918 7061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: attempting to promote NON_VOTER 681be2f83c664d9788066b058eae4063 to VOTER
I20260924 19:36:54.270124 7061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:54.270561 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 LEARNER]: Refusing update from remote peer 0b95402c5e7d43098c928ceeeea67ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:36:54.270720 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 19:36:54.271152 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.270475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45278)
0924 19:36:54.271082 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.271125 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
I20260924 19:36:54.271517 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 681be2f83c664d9788066b058eae4063 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.271741 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 681be2f83c664d9788066b058eae4063 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.272439 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.272501 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:54.272539 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.272576 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:36:54.272542 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.272606 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:36:54.271495 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.271412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45280)
0924 19:36:54.271445 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.271466 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 19:36:54.272930 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0 -> Peer 441d9cc31fcb4f59a3e4797b9aba9fc5 (127.0.98.60:42337): Couldn't send request to peer 441d9cc31fcb4f59a3e4797b9aba9fc5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:36:54.276203 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.001s
I20260924 19:36:54.276710 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.276855 7153 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.276917 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.276942 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
format_stamp: "Formatted at 2026-09-24 19:36:54 on dist-test-slave-6kf3"
server_key: "392f58b14544e4c66be727941f2fa493"
server_key_iv: "05862e4f579cac195cae112a6e0e2dda"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.277740 7153 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:54.279094 7153 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:36:54.279157 7153 log.cc:1196] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:36:54.279327 7153 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Deleting consensus metadata
I20260924 19:36:54.279438 7153 master_runner.cc:315] Copying system tablet from 127.0.98.62:33879
I20260924 19:36:54.280046 7153 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33879
I20260924 19:36:54.286525 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.280439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.0.98.62:33879)
0924 19:36:54.280481 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.280483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.280521 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:36:54.280581 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.280582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.280592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.280643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.280935 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.281688 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.282699 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.282973 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.283037 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.284024 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.284026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.284026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.284033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.285471 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.285823 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.285824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.285824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.285880 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.286055 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.286056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.286056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.286173 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.286230 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.286370 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.286446 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:36:54.286531 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.280286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:49858)
0924 19:36:54.280403 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.280473 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.280536 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.280649 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.280650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.280824 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.280868 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.280870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.281773 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.281774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.282615 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.282618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.283045 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.283046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.283102 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.283553 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.283558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.283921 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.285551 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.285552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.285553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.285755 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.285937 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.285937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.285938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.286003 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.286230 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.286230 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.286277 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.286344 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.286442 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260924 19:36:54.287058 6521 tablet_copy_service.cc:140] P 0b95402c5e7d43098c928ceeeea67ea0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 441d9cc31fcb4f59a3e4797b9aba9fc5 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49858)
I20260924 19:36:54.287109 6521 tablet_copy_service.cc:161] P 0b95402c5e7d43098c928ceeeea67ea0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 441d9cc31fcb4f59a3e4797b9aba9fc5 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49858: session id = 441d9cc31fcb4f59a3e4797b9aba9fc5-00000000000000000000000000000000
I20260924 19:36:54.287624 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:36:54.288115 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:36:54.289263 7153 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: tablet copy: Starting download of 0 data blocks...
I20260924 19:36:54.289413 7153 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: tablet copy: Starting download of 1 WAL segments...
I20260924 19:36:54.290163 7153 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:36:54.291040 6521 tablet_copy_service.cc:342] P 0b95402c5e7d43098c928ceeeea67ea0: Request end of tablet copy session 441d9cc31fcb4f59a3e4797b9aba9fc5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49858
I20260924 19:36:54.291088 6521 tablet_copy_service.cc:433] P 0b95402c5e7d43098c928ceeeea67ea0: ending tablet copy session 441d9cc31fcb4f59a3e4797b9aba9fc5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 441d9cc31fcb4f59a3e4797b9aba9fc5
I20260924 19:36:54.293282 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:54.294342 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.296803 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.296948 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.299549 7153 server_base.cc:1034] running on GCE node
I20260924 19:36:54.299639 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:54.299759 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:54.301003 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790278614300995 us; error 26 us; skew 500 ppm
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:54.302443 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 19:36:54.302687 7153 webserver.cc:460] Webserver started at http://127.0.98.60:38647/ using document root <none> and password file <none>
I20260924 19:36:54.302793 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.302824 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.303473 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:36:54.307997 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.308156 7153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260924 19:36:54.308209 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5"
format_stamp: "Formatted at 2026-09-24 19:36:54 on dist-test-slave-6kf3"
server_key: "392f58b14544e4c66be727941f2fa493"
server_key_iv: "05862e4f579cac195cae112a6e0e2dda"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.308264 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.318423 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.320080 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.322958 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42337
I20260924 19:36:54.323071 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42337 every 8 connection(s)
I20260924 19:36:54.323251 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:36:54.324666 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:36:54.325183 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Bootstrap starting.
I20260924 19:36:54.327904 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:36:54.328132 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: Bootstrap complete.
I20260924 19:36:54.328305 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:54.328394 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441d9cc31fcb4f59a3e4797b9aba9fc5, State: Initialized, Role: LEARNER
I20260924 19:36:54.328429 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:54.328594 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:36:54.328766 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.328927 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:36:54.330984 7335 catalog_manager.cc:1259] Loaded cluster ID: e2ab2bc2c5394b9f92837f8cfd4b114b
I20260924 19:36:54.331063 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: loading cluster ID for follower catalog manager: success
I20260924 19:36:54.332244 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: acquiring CA information for follower catalog manager: success
I20260924 19:36:54.332605 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:36:54.356489 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:54.356688 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:54.356717 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:54.357797 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:54.357836 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:54.357869 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:36:54.359328 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45235
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:54.359546 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:54.359740 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:54.364310 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.364310 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.364348 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.364635 7246 server_base.cc:1034] running on GCE node
I20260924 19:36:54.364828 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:54.365036 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:54.366168 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790278614366152 us; error 32 us; skew 500 ppm
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:54.368429 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:36:54.368794 7246 webserver.cc:460] Webserver started at http://127.0.98.1:45235/ using document root <none> and password file <none>
I20260924 19:36:54.368996 7246 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.369050 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.370247 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:54.370741 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.370864 7246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.370934 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "3bbbed92bd404586a62f3dfe069860a3"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "120ceca457e407d8c951b1e3d729916d"
server_key_iv: "81e6c4c492caa96df886d53d1eec96b6"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.371239 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.379189 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.380770 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:54.380888 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.381070 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:54.381322 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:54.381353 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.381378 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:54.381410 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.386943 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42241
I20260924 19:36:54.387001 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42241 every 8 connection(s)
I20260924 19:36:54.387283 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.396435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 7246
I20260924 19:36:54.396582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6697
I20260924 19:36:54.397735 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.388303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37183 (local address 127.0.98.62:33879)
0924 19:36:54.388352 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.388354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.388724 (+ 370us) server_negotiation.cc:407] Connection header received
0924 19:36:54.389253 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.389254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.389266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.389317 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.390121 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.390883 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.391986 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392172 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392229 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.394839 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.394840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.394840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.394847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.396267 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.396633 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.396634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.396634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.396681 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.397212 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.397213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.397214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.397347 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.397417 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.397578 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.397650 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:36:54.397756 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.388383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47123 (local address 127.0.98.61:41695)
0924 19:36:54.388460 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.388462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.388729 (+ 267us) server_negotiation.cc:407] Connection header received
0924 19:36:54.389314 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.389316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.389327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.389378 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.390148 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.390631 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.391892 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392198 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392252 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.394441 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.394442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.394443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.394449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.395980 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.396374 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.396375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.396375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.396430 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.397097 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.397098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.397099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.397234 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.397272 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.397596 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.397671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:36:54.397907 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.388232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:37183)
0924 19:36:54.388671 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.388679 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.389160 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.389685 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.389685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.389858 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.390046 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.390050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.390959 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.390960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.391774 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.391776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392276 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.392346 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.393057 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.393061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.394758 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.396330 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.396331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.396332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.396560 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.397093 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.397093 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.397094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.397147 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.397423 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.397424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.397455 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.397557 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.397703 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":6,"thread_start_us":82,"threads_started":1}
I20260924 19:36:54.398022 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.387904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.98.1:47123)
0924 19:36:54.388353 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.388363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.389156 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.389423 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.389427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.389794 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.390047 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.390053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.390720 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.390720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.391774 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.391776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392297 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.392355 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.392874 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.392884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.394358 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.396062 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.396068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.396071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.396325 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.396485 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.396485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.396486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.396586 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.397306 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.397311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.397433 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.397859 (+ 426us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.397939 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":8,"thread_start_us":112,"threads_started":1}
I20260924 19:36:54.398378 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41695
I20260924 19:36:54.398466 7466 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.398633 7466 heartbeater.cc:511] Master 127.0.98.61:41695 requested a full tablet report, sending...
I20260924 19:36:54.399123 7099 ts_manager.cc:194] Registered new tserver with Master: 3bbbed92bd404586a62f3dfe069860a3 (127.0.98.1:42241)
I20260924 19:36:54.401739 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.387898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49489 (local address 127.0.98.60:42337)
0924 19:36:54.388045 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.388047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.388723 (+ 676us) server_negotiation.cc:407] Connection header received
0924 19:36:54.389431 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.389432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.389462 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.389524 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.390155 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.391068 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.391815 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392139 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392197 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.395418 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.395419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.395420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.395427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.400762 (+ 5335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.401104 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.401105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.401106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.401169 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.401328 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.401328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.401328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.401424 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.401484 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.401611 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.401675 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260924 19:36:54.401798 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.387903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.98.1:49489)
0924 19:36:54.388070 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.388109 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.389171 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.389547 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.389548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.389854 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.390047 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.390053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.391131 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.391132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.391744 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.391746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.392219 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.392220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.392346 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.392982 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.392986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.395328 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.400844 (+ 5516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.400845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.400845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.401043 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.401221 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.401221 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.401222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.401278 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.401489 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.401489 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.401530 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.401591 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.401684 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260924 19:36:54.402160 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42337
I20260924 19:36:54.402204 7464 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.402261 7464 heartbeater.cc:511] Master 127.0.98.60:42337 requested a full tablet report, sending...
I20260924 19:36:54.402626 7285 ts_manager.cc:194] Registered new tserver with Master: 3bbbed92bd404586a62f3dfe069860a3 (127.0.98.1:42241)
I20260924 19:36:54.402688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37301
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33879
--webserver_port=37017
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:36:54.402761 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:54.402801 7465 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.402851 7465 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:54.403095 6500 ts_manager.cc:194] Re-registered known tserver with Master: 3bbbed92bd404586a62f3dfe069860a3 (127.0.98.1:42241)
I20260924 19:36:54.403486 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37183
W20260924 19:36:54.482350 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:54.482546 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:54.482568 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:54.483700 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:54.483740 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:54.483767 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:36:54.485224 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37017
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:54.485433 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:54.485653 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:54.490350 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.490435 7472 server_base.cc:1034] running on GCE node
W20260924 19:36:54.490368 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.490578 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.490782 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:54.491032 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:54.492165 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790278614492151 us; error 30 us; skew 500 ppm
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:54.494465 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:36:54.494864 7472 webserver.cc:460] Webserver started at http://127.0.98.2:37017/ using document root <none> and password file <none>
I20260924 19:36:54.495085 7472 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.495149 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.496407 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:54.496886 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.497035 7472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.497110 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "31af10e1de594a5c91e190d7b9d2f1c1"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "4f5af51ac435eeb9b705823e82d16045"
server_key_iv: "b4b1832b5479b947523205a1377ddd7e"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.497414 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.523495 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.525633 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:54.525764 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.525987 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:54.526276 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:54.526309 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.526335 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:54.526360 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.532204 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37301
I20260924 19:36:54.532260 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37301 every 8 connection(s)
I20260924 19:36:54.532573 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:36:54.537222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 7472
I20260924 19:36:54.537375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6831
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.542725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36729
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33879
--webserver_port=36475
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:36:54.543777 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34557 (local address 127.0.98.62:33879)
0924 19:36:54.533748 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.533749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.533759 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:54.534155 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.534157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.534170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.534215 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.535095 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.535857 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.537858 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.538063 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.538125 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.540227 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.540228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.540228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.540236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.542126 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542654 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.542655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.542656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.542713 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.543083 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.543084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.543085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.543224 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.543296 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.543604 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.543687 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260924 19:36:54.544312 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:34557)
0924 19:36:54.533413 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.533455 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.534058 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.534616 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.534617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.534802 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.535008 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.535012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.535965 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.535966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.536878 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.536880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.538133 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.538134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.538211 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.538651 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.538661 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.540124 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.542204 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.542205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.542568 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.542784 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.542785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.543010 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.543291 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.543295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.543452 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.543872 (+ 420us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.544074 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260924 19:36:54.544318 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36171 (local address 127.0.98.60:42337)
0924 19:36:54.533668 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.533670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.533736 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:36:54.534271 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.534273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.534287 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.534338 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.535110 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.535600 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.536959 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.537220 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.537291 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.540732 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.540734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.540734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.540741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.542117 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542652 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.542653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.542654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.542710 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.543494 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.543495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.543496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.543624 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.543710 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.544134 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.544226 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:36:54.544718 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:54.544812 7602 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.545007 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.98.2:36171)
0924 19:36:54.533676 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.533685 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.534053 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.534367 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.534371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.534807 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.535009 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.535015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.535694 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.535694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.536879 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.536880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.537299 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.537300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.538515 (+ 1215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.538875 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.538880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.540420 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.542193 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.542203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.542567 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.542778 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.542779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.542779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.543434 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.543735 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.543735 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.543776 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.544709 (+ 933us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.544885 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":9,"thread_start_us":63,"threads_started":1}
I20260924 19:36:54.545009 7602 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:54.545596 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42337
I20260924 19:36:54.545670 7601 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.545769 7601 heartbeater.cc:511] Master 127.0.98.60:42337 requested a full tablet report, sending...
I20260924 19:36:54.545817 6500 ts_manager.cc:194] Re-registered known tserver with Master: 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
I20260924 19:36:54.546293 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34557
I20260924 19:36:54.546406 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52975 (local address 127.0.98.61:41695)
0924 19:36:54.533738 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.533740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.533750 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:36:54.534384 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.534385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.534395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.534475 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.535116 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.535851 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.540664 (+ 4813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.540940 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.541005 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.543786 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.543787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.543788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.543795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.545359 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.545684 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.545685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.545685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.545739 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.545946 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.545947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.545947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.546046 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.546098 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.546252 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.546321 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:36:54.546623 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.533249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.98.2:52975)
0924 19:36:54.533415 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.533455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.534041 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.534479 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.534480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.534807 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.535009 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.535015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.536976 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.536977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.540566 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.540568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.541035 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.541036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.541356 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.542250 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.542254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.543700 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.545441 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.545442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.545442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.545603 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.545806 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.545807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.545807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.545860 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.546102 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.546103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.546142 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.546226 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.546314 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":29,"thread_start_us":91,"threads_started":1}
I20260924 19:36:54.546716 7285 ts_manager.cc:194] Registered new tserver with Master: 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
I20260924 19:36:54.546908 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41695
I20260924 19:36:54.546969 7603 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.547024 7603 heartbeater.cc:511] Master 127.0.98.61:41695 requested a full tablet report, sending...
I20260924 19:36:54.547289 7099 ts_manager.cc:194] Registered new tserver with Master: 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
W20260924 19:36:54.622018 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:36:54.622201 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:36:54.622221 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:36:54.623265 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:36:54.623296 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:36:54.623319 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:36:54.624672 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36475
--tserver_master_addrs=127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:36:54.624836 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:36:54.625012 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:36:54.629586 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.629693 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:36:54.629562 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:36:54.629668 7608 server_base.cc:1034] running on GCE node
I20260924 19:36:54.629927 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:36:54.630134 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:36:54.631296 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790278614631278 us; error 31 us; skew 500 ppm
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:36:54.633508 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:36:54.633903 7608 webserver.cc:460] Webserver started at http://127.0.98.3:36475/ using document root <none> and password file <none>
I20260924 19:36:54.634091 7608 fs_manager.cc:359] Metadata directory not provided
I20260924 19:36:54.634153 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:36:54.635340 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:36:54.635810 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.635977 7608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.636058 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "551edc6f070f45b8827773515d61c575"
format_stamp: "Formatted at 2026-09-24 19:36:53 on dist-test-slave-6kf3"
server_key: "759a8948f8d21d73365560a4be98d06d"
server_key_iv: "ae4d26f405764c975265c5da65e23fde"
server_key_version: "encryptionkey@0"
I20260924 19:36:54.636368 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:36:54.646484 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:36:54.648815 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:36:54.648972 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:36:54.649173 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:36:54.649478 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:36:54.649523 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.649550 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:36:54.649572 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:36:54.655687 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36729
I20260924 19:36:54.655746 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36729 every 8 connection(s)
I20260924 19:36:54.656090 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:36:54.657634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 7608
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:36:54 dist-test-slave-6kf3 krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:36:54.668515 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.657012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42943 (local address 127.0.98.61:41695)
0924 19:36:54.657056 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.657058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.657066 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.657516 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.657517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.657527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.657565 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.658671 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659446 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661038 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661330 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661367 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.665450 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.665451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.665452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.665458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.666919 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.667363 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.667364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.667365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.667431 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.667699 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.667700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.667700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.667819 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.667860 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.668337 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.668412 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:36:54.668867 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.656965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43327 (local address 127.0.98.62:33879)
0924 19:36:54.657027 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.657029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.658381 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 19:36:54.658497 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.658498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.658509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.658538 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.658957 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659654 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661019 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661295 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661333 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.665026 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.665027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.665028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.665035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.667608 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.668057 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.668058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.668058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.668141 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.668324 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.668325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.668325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.668438 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.668487 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.668719 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.668787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:36:54.669350 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.656725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:43327)
0924 19:36:54.658311 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.658321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.658409 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.658599 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.658599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.658791 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.658853 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.658855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659737 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.659738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.660941 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.660944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661407 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.661558 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.662181 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.662190 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.664913 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.667685 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.667686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.667686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.667862 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.668206 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.668207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.668207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.668258 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.668517 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.668518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.668554 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.668988 (+ 434us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.669114 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"mutex_wait_us":31,"thread_start_us":70,"threads_started":1}
I20260924 19:36:54.669546 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.659686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.0.98.60:42337)
0924 19:36:54.661938 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.661939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.662043 (+ 104us) server_negotiation.cc:407] Connection header received
0924 19:36:54.662063 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.662064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.662075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.663224 (+ 1149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.663258 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.663978 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.664911 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.665196 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.665244 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.666405 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.666406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.666406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.666413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.668180 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.668549 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.668550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.668551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.668613 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.668800 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.668801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.668801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.668925 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.669013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.669368 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.669454 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2224,"thread_start_us":45,"threads_started":1}
I20260924 19:36:54.669350 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.656724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41695 (local address 127.0.98.3:42943)
0924 19:36:54.656886 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.656927 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.657421 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.657643 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.657647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.658405 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.658563 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.658568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659571 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.659572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.660938 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.660940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661445 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.661558 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.662339 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.662343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.665347 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.667002 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.667009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.667011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.667280 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.667503 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.667503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.667504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.667628 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.667902 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.667909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.668187 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.668812 (+ 625us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.669157 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":123,"thread_start_us":92,"threads_started":1}
I20260924 19:36:54.669971 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33879
I20260924 19:36:54.670069 7738 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.670251 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41695
I20260924 19:36:54.670339 7739 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.670488 7739 heartbeater.cc:511] Master 127.0.98.61:41695 requested a full tablet report, sending...
I20260924 19:36:54.670712 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.661970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.0.98.61:41695)
0924 19:36:54.663364 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.663366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.663372 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:36:54.663390 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.663390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.663400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.663913 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.664446 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.665180 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.666143 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.666427 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.666667 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.667642 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.667643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.667644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.667650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.669206 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.669537 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.669538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.669538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.669587 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.670067 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.670068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.670068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.670183 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.670517 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.670556 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.670627 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":48,"threads_started":1}
I20260924 19:36:54.670254 7738 heartbeater.cc:511] Master 127.0.98.62:33879 requested a full tablet report, sending...
I20260924 19:36:54.670943 7099 ts_manager.cc:194] Registered new tserver with Master: 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729)
I20260924 19:36:54.670955 6500 ts_manager.cc:194] Re-registered known tserver with Master: 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729)
I20260924 19:36:54.671386 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43327
I20260924 19:36:54.672655 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.658799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.0.98.62:33879)
0924 19:36:54.659901 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.659903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.660081 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:36:54.660102 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.660103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.660114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.660161 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.660920 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661622 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.662954 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.663239 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.663297 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.669759 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.669760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.669761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.669768 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.671484 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.671999 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.672000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.672000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.672053 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.672249 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.672250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.672250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.672351 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.672419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.672535 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.672594 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":558,"threads_started":1}
I20260924 19:36:54.673682 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.656705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60573 (local address 127.0.98.60:42337)
0924 19:36:54.657015 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.657017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.657179 (+ 162us) server_negotiation.cc:407] Connection header received
0924 19:36:54.657506 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.657508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.657520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.657553 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.658671 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659445 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661039 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661335 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661369 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.671323 (+ 9954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.671324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.671324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.671331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.672704 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.673013 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.673013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.673014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.673057 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.673227 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.673228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.673228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.673329 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.673382 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.673518 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.673593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260924 19:36:54.673707 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.656725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.98.3:60573)
0924 19:36:54.657125 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.657134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.657431 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.657643 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.657646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.658291 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:36:54.658563 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.658568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.659571 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.659572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.660938 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.660940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.661438 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.661439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.661558 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.662233 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:36:54.662237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:36:54.671226 (+ 8989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:36:54.672766 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.672767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.672767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.672950 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.673111 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:36:54.673111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:36:54.673112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:36:54.673167 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:36:54.673387 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:36:54.673388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:36:54.673430 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.673488 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.673595 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":111,"thread_start_us":90,"threads_started":1}
I20260924 19:36:54.674222 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42337
I20260924 19:36:54.674269 7737 heartbeater.cc:464] Registering TS with master...
I20260924 19:36:54.674327 7737 heartbeater.cc:511] Master 127.0.98.60:42337 requested a full tablet report, sending...
I20260924 19:36:54.674638 7285 ts_manager.cc:194] Registered new tserver with Master: 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729)
I20260924 19:36:54.674217 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:36:54.675899 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:36:54.682744 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.3:36729)
0924 19:36:54.679635 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.679640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.679648 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.679681 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.679684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.679699 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.679778 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.679877 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.680687 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.681767 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.682448 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.682503 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.682576 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.682639 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.682672 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":70,"threads_started":1}
I20260924 19:36:54.682734 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36729 (local address 127.0.0.1:47852)
0924 19:36:54.679359 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.679370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.679409 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.679774 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.679775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.679777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.679822 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.679823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.680750 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.680751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.681679 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.681681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.682512 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.682513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.682570 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.682574 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.682625 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.682665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 19:36:54.683436 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42241 (local address 127.0.0.1:59840)
0924 19:36:54.679858 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.679867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.679905 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.680195 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.680196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.680198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.680234 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.680235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.681561 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.681562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.682357 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.682359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.683222 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.683223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.683283 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.683288 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.683337 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.683371 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260924 19:36:54.683393 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 3571afdbb0fa4452916faace6abb342c (DEFAULT_TABLE table=test-table [id=0f9b0da64d4f4cde806ba60ab9c9946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:54.683593 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37301 (local address 127.0.0.1:44882)
0924 19:36:54.679469 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.679479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.679535 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.679796 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.679797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.679799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.679874 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.679876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.681161 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.681162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.682210 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.682212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.683342 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.683343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.683414 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.683418 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.683477 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.683524 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260924 19:36:54.683763 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3571afdbb0fa4452916faace6abb342c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:54.683856 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.0.98.1:42241)
0924 19:36:54.680046 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.680051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.680059 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.680096 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.680098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.680113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.680187 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.680296 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.681474 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.682447 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.683132 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.683196 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.683264 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.683740 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.683779 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":62,"threads_started":1}
I20260924 19:36:54.684033 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.679317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.0.98.2:37301)
0924 19:36:54.679468 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.679474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.679523 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:36:54.679689 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.679692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.679707 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.679963 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.680025 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.681082 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.682288 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.683275 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.683339 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.683400 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.683706 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.683873 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260924 19:36:54.684408 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 3571afdbb0fa4452916faace6abb342c (DEFAULT_TABLE table=test-table [id=0f9b0da64d4f4cde806ba60ab9c9946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:54.684669 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3571afdbb0fa4452916faace6abb342c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:54.686544 7763 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Bootstrap starting.
I20260924 19:36:54.687201 7763 tablet_bootstrap.cc:654] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:54.687196 7764 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Bootstrap starting.
I20260924 19:36:54.687510 7763 log.cc:824] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:54.687718 7764 tablet_bootstrap.cc:654] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:54.687713 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 3571afdbb0fa4452916faace6abb342c (DEFAULT_TABLE table=test-table [id=0f9b0da64d4f4cde806ba60ab9c9946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:36:54.688060 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3571afdbb0fa4452916faace6abb342c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:36:54.688263 7763 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: No bootstrap required, opened a new log
I20260924 19:36:54.688326 7763 ts_tablet_manager.cc:1397] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:36:54.688510 7764 log.cc:824] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:54.689155 7764 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: No bootstrap required, opened a new log
I20260924 19:36:54.689211 7764 ts_tablet_manager.cc:1397] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:36:54.690694 7763 raft_consensus.cc:348] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.690860 7763 raft_consensus.cc:374] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:54.690894 7763 raft_consensus.cc:729] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551edc6f070f45b8827773515d61c575, State: Initialized, Role: FOLLOWER
I20260924 19:36:54.690827 7764 raft_consensus.cc:348] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.690979 7764 raft_consensus.cc:374] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:54.691016 7764 raft_consensus.cc:729] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbbed92bd404586a62f3dfe069860a3, State: Initialized, Role: FOLLOWER
I20260924 19:36:54.691025 7763 consensus_queue.cc:260] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.691125 7764 consensus_queue.cc:260] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.691250 7763 ts_tablet_manager.cc:1428] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:36:54.691357 7764 ts_tablet_manager.cc:1428] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:36:54.691371 7738 heartbeater.cc:503] Master 127.0.98.62:33879 was elected leader, sending a full tablet report...
I20260924 19:36:54.691808 7465 heartbeater.cc:503] Master 127.0.98.62:33879 was elected leader, sending a full tablet report...
I20260924 19:36:54.691560 7767 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Bootstrap starting.
I20260924 19:36:54.693081 7767 tablet_bootstrap.cc:654] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:36:54.693336 7767 log.cc:824] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:36:54.694214 7767 tablet_bootstrap.cc:492] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: No bootstrap required, opened a new log
I20260924 19:36:54.694263 7767 ts_tablet_manager.cc:1397] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:36:54.696471 7767 raft_consensus.cc:348] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.696595 7767 raft_consensus.cc:374] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:36:54.696616 7767 raft_consensus.cc:729] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31af10e1de594a5c91e190d7b9d2f1c1, State: Initialized, Role: FOLLOWER
I20260924 19:36:54.696725 7767 consensus_queue.cc:260] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.696957 7767 ts_tablet_manager.cc:1428] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:36:54.697018 7602 heartbeater.cc:503] Master 127.0.98.62:33879 was elected leader, sending a full tablet report...
I20260924 19:36:54.758460 7769 raft_consensus.cc:482] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:36:54.758585 7769 raft_consensus.cc:504] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.759115 7769 leader_election.cc:288] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729), 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
I20260924 19:36:54.761703 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.759258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36729 (local address 127.0.98.1:48613)
0924 19:36:54.759314 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.759324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.759361 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.759494 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.759495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.759496 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.759575 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.759577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.760283 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.760284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.761001 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.761002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.761495 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.761561 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.761566 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.761612 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.761650 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":36}
I20260924 19:36:54.761722 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.759326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48613 (local address 127.0.98.3:36729)
0924 19:36:54.759370 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.759372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.759383 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:36:54.759412 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.759413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.759425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.759481 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.759638 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.760216 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761083 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.761433 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.761496 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.761621 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.761659 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:36:54.761848 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.759281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37301 (local address 127.0.98.1:60663)
0924 19:36:54.759314 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.759324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.759361 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.759565 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.759565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.759567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.759623 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.759624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.760245 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.760246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.760998 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.761000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.761602 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.761691 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.761695 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.761747 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.761778 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":1}
I20260924 19:36:54.761870 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.759393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60663 (local address 127.0.98.2:37301)
0924 19:36:54.759450 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.759451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.759458 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:54.759482 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.759483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.759495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.759550 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.759687 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.760184 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761079 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.761528 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.761593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.761603 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.761763 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.761807 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:36:54.762010 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3571afdbb0fa4452916faace6abb342c" candidate_uuid: "3bbbed92bd404586a62f3dfe069860a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551edc6f070f45b8827773515d61c575" is_pre_election: true
I20260924 19:36:54.762176 7691 raft_consensus.cc:2454] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bbbed92bd404586a62f3dfe069860a3 in term 0.
I20260924 19:36:54.762176 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3571afdbb0fa4452916faace6abb342c" candidate_uuid: "3bbbed92bd404586a62f3dfe069860a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" is_pre_election: true
I20260924 19:36:54.762286 7555 raft_consensus.cc:2454] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bbbed92bd404586a62f3dfe069860a3 in term 0.
I20260924 19:36:54.762383 7351 leader_election.cc:302] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bbbed92bd404586a62f3dfe069860a3, 551edc6f070f45b8827773515d61c575; no voters:
I20260924 19:36:54.762563 7769 raft_consensus.cc:2781] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:36:54.762619 7769 raft_consensus.cc:482] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:36:54.762634 7769 raft_consensus.cc:3037] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:54.763278 7769 raft_consensus.cc:504] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.763392 7769 leader_election.cc:288] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [CANDIDATE]: Term 1 election: Requested vote from peers 551edc6f070f45b8827773515d61c575 (127.0.98.3:36729), 31af10e1de594a5c91e190d7b9d2f1c1 (127.0.98.2:37301)
I20260924 19:36:54.763545 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3571afdbb0fa4452916faace6abb342c" candidate_uuid: "3bbbed92bd404586a62f3dfe069860a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31af10e1de594a5c91e190d7b9d2f1c1"
I20260924 19:36:54.763564 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3571afdbb0fa4452916faace6abb342c" candidate_uuid: "3bbbed92bd404586a62f3dfe069860a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551edc6f070f45b8827773515d61c575"
I20260924 19:36:54.763629 7691 raft_consensus.cc:3037] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:54.763629 7555 raft_consensus.cc:3037] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:36:54.764494 7691 raft_consensus.cc:2454] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bbbed92bd404586a62f3dfe069860a3 in term 1.
I20260924 19:36:54.764494 7555 raft_consensus.cc:2454] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bbbed92bd404586a62f3dfe069860a3 in term 1.
I20260924 19:36:54.764648 7352 leader_election.cc:302] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31af10e1de594a5c91e190d7b9d2f1c1, 3bbbed92bd404586a62f3dfe069860a3; no voters:
I20260924 19:36:54.764720 7769 raft_consensus.cc:2781] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:36:54.764878 7769 raft_consensus.cc:686] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3bbbed92bd404586a62f3dfe069860a3, State: Running, Role: LEADER
I20260924 19:36:54.764971 7769 consensus_queue.cc:237] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } }
I20260924 19:36:54.765772 6500 catalog_manager.cc:5495] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bbbed92bd404586a62f3dfe069860a3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3bbbed92bd404586a62f3dfe069860a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bbbed92bd404586a62f3dfe069860a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42241 } health_report { overall_health: HEALTHY } } }
I20260924 19:36:54.773164 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.770913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.1:42241)
0924 19:36:54.770952 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.770954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.770962 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:36:54.771022 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.771023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.771038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.771089 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:36:54.771175 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.771638 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.772429 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.772609 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.772652 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.772794 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:36:54.772919 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:36:54.772991 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.773080 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.773099 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:36:54.774735 7555 raft_consensus.cc:1261] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 1 FOLLOWER]: Refusing update from remote peer 3bbbed92bd404586a62f3dfe069860a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:54.774735 7691 raft_consensus.cc:1261] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 1 FOLLOWER]: Refusing update from remote peer 3bbbed92bd404586a62f3dfe069860a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:36:54.775055 7772 consensus_queue.cc:1035] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31af10e1de594a5c91e190d7b9d2f1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:54.775170 7769 consensus_queue.cc:1035] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "551edc6f070f45b8827773515d61c575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:36:54.777429 7775 mvcc.cc:205] Tried to move back new op lower bound from 7332981206115123200 to 7332981206078070784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981206115123200}]
I20260924 19:36:54.778976 7776 mvcc.cc:205] Tried to move back new op lower bound from 7332981206115123200 to 7332981206078070784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981206115123200}]
I20260924 19:36:54.779101 7777 mvcc.cc:205] Tried to move back new op lower bound from 7332981206115123200 to 7332981206078070784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981206115123200 or (T in {7332981206115123200})}]
I20260924 19:36:54.780391 6500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:54.780483 6500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:36:54.781546 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.778695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42337 (local address 127.0.0.1:45304)
0924 19:36:54.778751 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:54.778762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:36:54.778798 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:36:54.778943 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:36:54.778944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:36:54.778946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:36:54.778991 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.778993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.779791 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.779792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.780608 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:36:54.780610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.781286 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.781287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:36:54.781357 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.781361 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:36:54.781417 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:36:54.781474 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:36:54.781569 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.778780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.98.60:42337)
0924 19:36:54.778817 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.778818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.778827 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:36:54.778845 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.778846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.778867 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.778929 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:36:54.779088 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.779717 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.780680 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.781219 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.781287 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.781359 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.781460 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.781504 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:54.781994 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260924 19:36:54.781988 6500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b95402c5e7d43098c928ceeeea67ea0: Sending DeleteTablet for 3 replicas of tablet 3571afdbb0fa4452916faace6abb342c
I20260924 19:36:54.782284 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 3571afdbb0fa4452916faace6abb342c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59840
I20260924 19:36:54.782392 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 3571afdbb0fa4452916faace6abb342c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260924 19:36:54.782617 7787 tablet_replica.cc:330] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: stopping tablet replica
I20260924 19:36:54.782747 7787 raft_consensus.cc:2229] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:54.782754 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b95402c5e7d43098c928ceeeea67ea0. Latest consensus state: current_term: 1 leader_uuid: "0b95402c5e7d43098c928ceeeea67ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } } }
I20260924 19:36:54.782843 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441d9cc31fcb4f59a3e4797b9aba9fc5 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:36:54.782992 7787 raft_consensus.cc:2258] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:54.783454 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3571afdbb0fa4452916faace6abb342c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:36:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47852
I20260924 19:36:54.783674 7787 ts_tablet_manager.cc:1905] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:54.784030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6469
I20260924 19:36:54.785341 7787 ts_tablet_manager.cc:1918] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:54.785478 7787 log.cc:1196] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/3571afdbb0fa4452916faace6abb342c
I20260924 19:36:54.785779 7787 ts_tablet_manager.cc:1939] T 3571afdbb0fa4452916faace6abb342c P 31af10e1de594a5c91e190d7b9d2f1c1: Deleting consensus metadata
I20260924 19:36:54.788110 7790 tablet_replica.cc:330] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: stopping tablet replica
I20260924 19:36:54.788221 7790 raft_consensus.cc:2229] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:36:54.788331 7790 raft_consensus.cc:2258] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:54.788800 7790 ts_tablet_manager.cc:1905] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:54.790138 7790 ts_tablet_manager.cc:1918] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:54.790251 7790 log.cc:1196] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/3571afdbb0fa4452916faace6abb342c
W20260924 19:36:54.790350 7270 connection.cc:535] server connection from 127.0.0.1:45304 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:36:54.790473 7489 connection.cc:535] server connection from 127.0.0.1:44882 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:36:54.790788 7790 ts_tablet_manager.cc:1939] T 3571afdbb0fa4452916faace6abb342c P 551edc6f070f45b8827773515d61c575: Deleting consensus metadata
W20260924 19:36:54.791177 7627 connection.cc:441] server connection from 127.0.0.1:47852 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47852 (request call id 1) could send its response
I20260924 19:36:54.792017 7785 tablet_replica.cc:330] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: stopping tablet replica
I20260924 19:36:54.792250 7785 raft_consensus.cc:2229] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:36:54.792397 7785 raft_consensus.cc:2258] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:36:54.793727 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.783241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.98.60:42337)
0924 19:36:54.783281 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.783283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.783290 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:36:54.783308 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.783309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.783318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.783366 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.784060 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.784688 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.785688 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.785959 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.786001 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.791236 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.791238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.791238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.791245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.792707 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.793054 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.793055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.793056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.793117 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.793271 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.793272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.793272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.793380 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.793486 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.793573 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.793642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:36:54.796340 7785 ts_tablet_manager.cc:1905] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:36:54.797881 7785 ts_tablet_manager.cc:1918] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:36:54.797952 7785 log.cc:1196] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/3571afdbb0fa4452916faace6abb342c
I20260924 19:36:54.798185 7785 ts_tablet_manager.cc:1939] T 3571afdbb0fa4452916faace6abb342c P 3bbbed92bd404586a62f3dfe069860a3: Deleting consensus metadata
W20260924 19:36:54.798606 7353 connection.cc:441] server connection from 127.0.0.1:59840 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59840 (request call id 1) could send its response
I20260924 19:36:54.799440 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.792140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.61:41695)
0924 19:36:54.792179 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.792181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.792707 (+ 526us) server_negotiation.cc:407] Connection header received
0924 19:36:54.792779 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.792780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.792793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.792842 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.793384 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.794156 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.795304 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.795557 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.795612 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.796799 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.796801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.796801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.796808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.798049 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.798355 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.798356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.798356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.798409 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.798816 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.798817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.798817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.798950 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.799056 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.799253 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.799345 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:54.801863 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:54.792367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.60:42337)
0924 19:36:54.792406 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:36:54.792408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:36:54.795413 (+ 3005us) server_negotiation.cc:407] Connection header received
0924 19:36:54.795454 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:36:54.795455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:36:54.795467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:36:54.795520 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:36:54.796165 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.796929 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.797844 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:36:54.798133 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:36:54.798178 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:36:54.799477 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:36:54.799479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:36:54.799479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:36:54.799485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:36:54.800945 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.801234 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.801235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.801235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.801291 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:36:54.801433 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:36:54.801434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:36:54.801434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:36:54.801554 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:36:54.801639 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:36:54.801699 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:36:54.801775 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:36:55.778792 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:55.778543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:38977)
0924 19:36:55.778660 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:55.778740 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
W20260924 19:36:55.778993 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:55.779500 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:55.779298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:34071)
0924 19:36:55.779399 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:55.779458 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
W20260924 19:36:55.779716 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:55.779738 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:55.779532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:35459)
0924 19:36:55.779641 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:55.779700 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
W20260924 19:36:55.779898 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:56.282048 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:36:56.282174 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:56.282413 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:36:56.282707 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:56.282530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:50304)
0924 19:36:56.282615 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:56.282670 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
W20260924 19:36:56.282821 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:56.283264 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:56.283168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:50308)
0924 19:36:56.283218 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:56.283244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:36:56.283355 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:56.283425 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:36:56.283528 7803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:36:56.779590 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:56.779323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:41077)
0924 19:36:56.779485 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:56.779535 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260924 19:36:56.780231 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:56.780034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:53541)
0924 19:36:56.780160 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:56.780196 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":44,"threads_started":1}
I20260924 19:36:56.780388 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:56.780216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:60503)
0924 19:36:56.780318 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:56.780349 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":42,"threads_started":1}
I20260924 19:36:57.780592 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:57.780327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:41235)
0924 19:36:57.780473 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:57.780515 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260924 19:36:57.780820 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:57.780571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:45115)
0924 19:36:57.780740 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:57.780777 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
I20260924 19:36:57.780920 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:57.780744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:37465)
0924 19:36:57.780847 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:57.780879 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260924 19:36:58.382715 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:36:58.382809 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:36:58.382926 7812 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:36:58.383244 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:58.383074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40398)
0924 19:36:58.383168 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:58.383204 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":43,"threads_started":1}
I20260924 19:36:58.383598 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:58.383517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40410)
0924 19:36:58.383556 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:58.383581 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:36:58.383667 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:36:58.383713 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:36:58.383782 7812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:36:58.781271 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:58.781001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:44067)
0924 19:36:58.781176 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:58.781218 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260924 19:36:58.781340 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:58.781134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:34583)
0924 19:36:58.781259 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:58.781295 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260924 19:36:58.781456 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:58.781230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:47389)
0924 19:36:58.781315 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:58.781338 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20260924 19:36:59.781890 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:59.781687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:53085)
0924 19:36:59.781805 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:59.781840 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":45,"threads_started":1}
I20260924 19:36:59.781890 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:59.781668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:47395)
0924 19:36:59.781814 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:59.781840 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260924 19:36:59.781991 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:36:59.781785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:59087)
0924 19:36:59.781900 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 19:36:59.781934 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260924 19:37:00.390408 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:37:00.390499 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:37:00.390692 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:37:00.391012 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:00.390837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40628)
0924 19:37:00.390925 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:00.390975 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":41,"threads_started":1}
I20260924 19:37:00.391415 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:00.391310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40644)
0924 19:37:00.391350 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:00.391391 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:37:00.391489 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:37:00.391534 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:37:00.391604 7821 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:37:00.782518 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:00.782263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:55573)
0924 19:37:00.782421 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:00.782466 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260924 19:37:00.782518 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:00.782263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:34713)
0924 19:37:00.782424 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:00.782464 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260924 19:37:00.782649 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:00.782377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:56705)
0924 19:37:00.782526 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:00.782558 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":50,"threads_started":1}
I20260924 19:37:01.783126 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:01.782913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:41345)
0924 19:37:01.783034 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:01.783071 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260924 19:37:01.783177 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:01.782953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:49479)
0924 19:37:01.783089 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:01.783131 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":39,"threads_started":1}
I20260924 19:37:01.783331 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:01.783084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:57465)
0924 19:37:01.783184 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:01.783231 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260924 19:37:02.783914 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:02.783662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:50017)
0924 19:37:02.783818 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:02.783862 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260924 19:37:02.784158 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:02.783663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:35027)
0924 19:37:02.784081 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:02.784115 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":68,"threads_started":1}
I20260924 19:37:02.784175 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:02.783659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:35483)
0924 19:37:02.784092 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:02.784135 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":74,"threads_started":1}
I20260924 19:37:03.784612 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:03.784376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:39179)
0924 19:37:03.784516 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:03.784560 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
I20260924 19:37:03.784759 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:03.784571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:39193)
0924 19:37:03.784682 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:03.784714 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 19:37:03.784775 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:03.784601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:58285)
0924 19:37:03.784709 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:03.784737 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260924 19:37:03.790416 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:37:03.790504 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:37:03.790683 7836 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:37:03.790967 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:03.790792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40902)
0924 19:37:03.790900 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:03.790934 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260924 19:37:03.791109 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260924 19:37:03.791430 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:03.791344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40914)
0924 19:37:03.791387 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:03.791411 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:37:03.791495 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:37:03.791534 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:37:03.791633 7836 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:37:04.785426 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:04.785090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:42681)
0924 19:37:04.785340 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:04.785376 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":191,"threads_started":1}
I20260924 19:37:04.785426 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:04.785179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:43509)
0924 19:37:04.785340 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:04.785376 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260924 19:37:04.785458 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:04.785090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:51797)
0924 19:37:04.785373 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:04.785411 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260924 19:37:05.786067 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:05.785832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:44283)
0924 19:37:05.785983 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:05.786021 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260924 19:37:05.786067 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:05.785832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:48305)
0924 19:37:05.786001 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:05.786028 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260924 19:37:05.786046 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:05.785834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:43807)
0924 19:37:05.785948 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:05.785989 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260924 19:37:06.786702 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:06.786494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:57359)
0924 19:37:06.786612 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:06.786650 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 19:37:06.786715 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:06.786515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:42385)
0924 19:37:06.786632 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:06.786668 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
I20260924 19:37:06.786901 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:06.786714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:42849)
0924 19:37:06.786837 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:06.786869 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260924 19:37:07.483451 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:37:07.483562 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:37:07.483729 7848 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:37:07.484068 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:07.483879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:41164)
0924 19:37:07.483982 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:07.484029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":104,"threads_started":1}
I20260924 19:37:07.484465 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:07.484331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:41176)
0924 19:37:07.484406 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:07.484441 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:37:07.484539 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:37:07.484588 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:37:07.484664 7848 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:37:07.787248 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:07.787020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:43187)
0924 19:37:07.787154 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:07.787193 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260924 19:37:07.787508 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:07.787096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:33619)
0924 19:37:07.787426 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:07.787463 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":66,"threads_started":1}
I20260924 19:37:07.787528 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:07.787266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:33095)
0924 19:37:07.787455 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:07.787488 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":49,"threads_started":1}
I20260924 19:37:08.788100 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:08.787733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:42635)
0924 19:37:08.788005 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:08.788043 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":63,"threads_started":1}
I20260924 19:37:08.788146 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:08.787865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:38603)
0924 19:37:08.788061 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:08.788099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":45,"threads_started":1}
I20260924 19:37:08.788244 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:08.787887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:54305)
0924 19:37:08.788161 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:08.788195 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":69,"threads_started":1}
I20260924 19:37:09.789120 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:09.788499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:58001)
0924 19:37:09.789039 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:09.789073 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":54,"threads_started":1}
I20260924 19:37:09.789121 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:09.788919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:40717)
0924 19:37:09.789022 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:09.789066 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260924 19:37:09.789207 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:09.788660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:33707)
0924 19:37:09.789124 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:09.789169 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":63,"threads_started":1}
I20260924 19:37:10.789821 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:10.789540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:57695)
0924 19:37:10.789654 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:10.789692 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 19:37:10.789865 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:10.789532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:55899)
0924 19:37:10.789763 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:10.789828 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260924 19:37:10.789898 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:10.789609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:53323)
0924 19:37:10.789817 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:10.789854 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":67,"threads_started":1}
I20260924 19:37:11.790429 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:11.790220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:54099)
0924 19:37:11.790337 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:11.790377 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 19:37:11.790535 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:11.790220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:60657)
0924 19:37:11.790453 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:11.790488 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":68,"threads_started":1}
I20260924 19:37:11.790719 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:11.790384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:44151)
0924 19:37:11.790652 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:11.790682 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":57,"threads_started":1}
I20260924 19:37:12.791069 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:12.790879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:51935)
0924 19:37:12.790999 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:12.791024 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":41,"threads_started":1}
I20260924 19:37:12.791077 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:12.790845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:50783)
0924 19:37:12.790981 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:12.791018 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 19:37:12.791230 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:12.791039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:46543)
0924 19:37:12.791134 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:12.791186 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260924 19:37:13.791643 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:13.791402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:48205)
0924 19:37:13.791544 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:13.791581 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260924 19:37:13.791750 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:13.791577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:46975)
0924 19:37:13.791670 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:13.791705 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
I20260924 19:37:13.791777 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:13.791575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:55681)
0924 19:37:13.791682 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:13.791711 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260924 19:37:14.418222 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b95402c5e7d43098c928ceeeea67ea0)
I20260924 19:37:14.418318 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b95402c5e7d43098c928ceeeea67ea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33879 } } peers { permanent_uuid: "681be2f83c664d9788066b058eae4063" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41695 } attrs { promote: false } } peers { permanent_uuid: "441d9cc31fcb4f59a3e4797b9aba9fc5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42337 } attrs { promote: true } }
I20260924 19:37:14.418433 7872 leader_election.cc:288] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879)
I20260924 19:37:14.418752 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:14.418569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40336)
0924 19:37:14.418679 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:14.418715 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
W20260924 19:37:14.418825 7085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260924 19:37:14.419209 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:14.419114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.0.1:40342)
0924 19:37:14.419161 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:14.419186 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:37:14.419269 7085 leader_election.cc:334] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b95402c5e7d43098c928ceeeea67ea0 (127.0.98.62:33879): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
I20260924 19:37:14.419305 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 681be2f83c664d9788066b058eae4063; no voters: 0b95402c5e7d43098c928ceeeea67ea0
I20260924 19:37:14.419363 7872 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681be2f83c664d9788066b058eae4063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:37:14.792321 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:14.792099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:50531)
0924 19:37:14.792228 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:14.792266 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260924 19:37:14.792366 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:14.792144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:43963)
0924 19:37:14.792287 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:14.792320 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260924 19:37:14.792452 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:14.792159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:51225)
0924 19:37:14.792381 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:14.792415 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":46,"threads_started":1}
I20260924 19:37:15.792901 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:15.792697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:38197)
0924 19:37:15.792821 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:15.792856 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260924 19:37:15.793001 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:15.792847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:50705)
0924 19:37:15.792940 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:15.792967 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260924 19:37:15.792999 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:15.792711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:33663)
0924 19:37:15.792850 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:15.792947 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260924 19:37:16.793497 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:16.793301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:58679)
0924 19:37:16.793433 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:16.793464 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260924 19:37:16.793480 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:16.793258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:51953)
0924 19:37:16.793389 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:16.793425 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260924 19:37:16.793520 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:16.793324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:60437)
0924 19:37:16.793447 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:16.793479 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":42,"threads_started":1}
I20260924 19:37:17.794041 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:17.793849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:54821)
0924 19:37:17.793963 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:17.793996 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260924 19:37:17.794037 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:17.793803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:56017)
0924 19:37:17.793944 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:17.793980 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260924 19:37:17.794070 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:17.793898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:50635)
0924 19:37:17.794006 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:17.794034 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":47,"threads_started":1}
I20260924 19:37:18.794615 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:18.794396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:56159)
0924 19:37:18.794523 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:18.794560 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 19:37:18.794688 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:18.794506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:37311)
0924 19:37:18.794628 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:18.794652 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":38,"threads_started":1}
I20260924 19:37:18.794688 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:18.794457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:54387)
0924 19:37:18.794613 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:18.794645 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":50,"threads_started":1}
I20260924 19:37:19.795274 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:19.795038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:49247)
0924 19:37:19.795187 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:19.795219 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":49,"threads_started":1}
I20260924 19:37:19.795326 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:19.795020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:58077)
0924 19:37:19.795263 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:19.795290 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":53,"threads_started":1}
I20260924 19:37:19.795388 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:19.795172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:53105)
0924 19:37:19.795313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:19.795343 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":55,"threads_started":1}
I20260924 19:37:20.796038 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:20.795830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:37499)
0924 19:37:20.795967 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:20.795999 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":52,"threads_started":1}
I20260924 19:37:20.796025 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:20.795780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:44171)
0924 19:37:20.795912 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:20.795970 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260924 19:37:20.796049 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:20.795780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:36831)
0924 19:37:20.795971 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:20.796002 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260924 19:37:21.796653 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:21.796373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:58789)
0924 19:37:21.796556 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:21.796610 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
I20260924 19:37:21.796658 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:21.796379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:37383)
0924 19:37:21.796565 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:21.796607 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":54,"threads_started":1}
I20260924 19:37:21.796727 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:21.796555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:43337)
0924 19:37:21.796652 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:21.796677 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260924 19:37:22.797179 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:22.796973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:39617)
0924 19:37:22.797090 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:22.797126 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260924 19:37:22.797219 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:22.797005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:34115)
0924 19:37:22.797135 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:22.797177 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":42,"threads_started":1}
I20260924 19:37:22.797402 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:22.797177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:50451)
0924 19:37:22.797337 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:22.797370 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":53,"threads_started":1}
I20260924 19:37:23.797868 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:23.797653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:45027)
0924 19:37:23.797779 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:23.797815 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260924 19:37:23.797890 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:23.797649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:41197)
0924 19:37:23.797828 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:23.797856 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260924 19:37:23.797914 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:23.797716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:34329)
0924 19:37:23.797832 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:23.797868 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":45,"threads_started":1}
I20260924 19:37:24.798553 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:24.798302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.3:44403)
0924 19:37:24.798492 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:24.798519 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":50,"threads_started":1}
I20260924 19:37:24.798550 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:24.798284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.2:32811)
0924 19:37:24.798463 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:24.798503 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":62,"threads_started":1}
I20260924 19:37:24.798578 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:24.798325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33879 (local address 127.0.98.1:48747)
0924 19:37:24.798434 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:24.798476 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:33879,127.0.98.61:41695,127.0.98.60:42337, num_attempts: 1)
I20260924 19:37:24.948817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 7246
I20260924 19:37:24.953708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 7472
I20260924 19:37:24.957484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 7608
I20260924 19:37:24.961102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 6965
I20260924 19:37:24.964546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 7153
2026-09-24T19:37:24Z chronyd exiting
I20260924 19:37:24.979996 392 test_util.cc:175] -----------------------------------------------
I20260924 19:37:24.980058 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278584646480-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32821 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:37:25 dist-test-slave-6kf3 krb5kdc[7910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:37:25 dist-test-slave-6kf3 krb5kdc[7910](info): set up 2 sockets
Sep 24 19:37:25 dist-test-slave-6kf3 krb5kdc[7910](info): commencing operation
krb5kdc: starting...
W20260924 19:37:25.916110 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:37:25 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 19:37:26.024010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41989
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:26.102041 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:26.102237 7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:26.102265 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:26.103355 7923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:26.103385 7923 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:37:26.103398 7923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:26.103410 7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:26.103420 7923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:26.103431 7923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:26.104811 7923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:26.104996 7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:26.105157 7923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:26.109987 7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:26.109953 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.109968 7923 server_base.cc:1034] running on GCE node
W20260924 19:37:26.109953 7931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.110479 7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:26.110742 7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:26.111904 7923 hybrid_clock.cc:648] HybridClock initialized: now 1790278646111896 us; error 30 us; skew 500 ppm
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:26.114455 7923 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 19:37:26.114885 7923 webserver.cc:460] Webserver started at http://127.0.98.62:40425/ using document root <none> and password file <none>
I20260924 19:37:26.115069 7923 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:26.115106 7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:26.115199 7923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:26.116240 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "1351a10d306d4c26ad4e8a5a6b0506de"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "984c5e964346c7396a14dfd789c867c3"
server_key_iv: "2cf80d89f0d4999455a4d189befe1384"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.116518 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "1351a10d306d4c26ad4e8a5a6b0506de"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "984c5e964346c7396a14dfd789c867c3"
server_key_iv: "2cf80d89f0d4999455a4d189befe1384"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.117753 7923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:26.118446 7938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.118680 7923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:26.118757 7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "1351a10d306d4c26ad4e8a5a6b0506de"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "984c5e964346c7396a14dfd789c867c3"
server_key_iv: "2cf80d89f0d4999455a4d189befe1384"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.118821 7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:26.145052 7923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:26.146860 7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:26.146966 7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:26.151240 7923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41989
I20260924 19:37:26.151300 8000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41989 every 8 connection(s)
I20260924 19:37:26.151582 7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:26.152199 8001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.154413 8001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Bootstrap starting.
I20260924 19:37:26.154953 8001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.155202 8001 log.cc:824] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:26.155834 8001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: No bootstrap required, opened a new log
I20260924 19:37:26.157845 8001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260924 19:37:26.157979 8001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.158011 8001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351a10d306d4c26ad4e8a5a6b0506de, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.158139 8001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260924 19:37:26.158213 8001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:26.158222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 7923
I20260924 19:37:26.158243 8001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:26.158293 8001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.158319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:37:26.158504 392 external_mini_cluster.cc:1428] Setting key b26674bc696ced13403ef5fda3e24de9
I20260924 19:37:26.158897 8001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260924 19:37:26.159060 8001 leader_election.cc:302] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1351a10d306d4c26ad4e8a5a6b0506de; no voters:
I20260924 19:37:26.159222 8001 leader_election.cc:288] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:37:26.159262 8006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:26.159420 8006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [term 1 LEADER]: Becoming Leader. State: Replica: 1351a10d306d4c26ad4e8a5a6b0506de, State: Running, Role: LEADER
I20260924 19:37:26.159452 8001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:26.160223 8006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260924 19:37:26.160833 8007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1351a10d306d4c26ad4e8a5a6b0506de. Latest consensus state: current_term: 1 leader_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } } }
I20260924 19:37:26.160909 8007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:26.161078 8008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1351a10d306d4c26ad4e8a5a6b0506de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } } }
I20260924 19:37:26.161123 8008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de [sys.catalog]: This master's current role is: LEADER
W20260924 19:37:26.161173 8020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:37:26.161224 8020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:37:26.161312 8022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:26.161464 8022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:26.163268 8022 catalog_manager.cc:1347] Generated new cluster ID: 9d47b0ca07d64d5b9fcdeae2bf703355
I20260924 19:37:26.163316 8022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.169174 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.158960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.98.62:41989)
0924 19:37:26.159147 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.159157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.159193 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:37:26.159318 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.159326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.159382 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.160046 (+ 664us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.160241 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.161684 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.162516 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.163055 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.163158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.165833 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.165837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.165838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.165848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.167643 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.168049 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.168052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.168052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.168153 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.168357 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.168357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.168358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.168620 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.168749 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.168847 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.168970 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260924 19:37:26.174369 8022 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:37:26.175069 8022 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:26.183050 8022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Generated new TSK 0
I20260924 19:37:26.183261 8022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 19:37:26.208856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:26.286993 8027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:26.287199 8027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:26.287230 8027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:26.288381 8027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:26.288422 8027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:26.288450 8027 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:37:26.288472 8027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:26.289923 8027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:26.290159 8027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:26.290356 8027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:26.294863 8035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.294970 8027 server_base.cc:1034] running on GCE node
W20260924 19:37:26.294888 8033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:26.294896 8032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.295238 8027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:26.295450 8027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:26.296571 8027 hybrid_clock.cc:648] HybridClock initialized: now 1790278646296561 us; error 27 us; skew 500 ppm
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:26.298827 8027 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:37:26.299219 8027 webserver.cc:460] Webserver started at http://127.0.98.1:34319/ using document root <none> and password file <none>
I20260924 19:37:26.299419 8027 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:26.299461 8027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:26.299578 8027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:26.300697 8027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "9acd5d4152974c1c869d455539af27eb"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "3f8e3f4d2a95449102d2c12afc140987"
server_key_iv: "2087b9f79b0bbd9093e19b43ac4666dd"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.301018 8027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9acd5d4152974c1c869d455539af27eb"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "3f8e3f4d2a95449102d2c12afc140987"
server_key_iv: "2087b9f79b0bbd9093e19b43ac4666dd"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.302203 8027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:37:26.302820 8042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.302958 8027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:26.303052 8027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "9acd5d4152974c1c869d455539af27eb"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "3f8e3f4d2a95449102d2c12afc140987"
server_key_iv: "2087b9f79b0bbd9093e19b43ac4666dd"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.303112 8027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:26.312677 8027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:26.314374 8027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:26.314500 8027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:26.314981 8027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:26.315021 8027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.315052 8027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:26.315074 8027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.321372 8027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35191
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.321419 8157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35191 every 8 connection(s)
I20260924 19:37:26.321764 8027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:37:26.323407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8027
I20260924 19:37:26.323513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:37:26.323702 392 external_mini_cluster.cc:1428] Setting key 15a4156700bf6ebb28f8eb00d63e23ad
I20260924 19:37:26.324625 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.315441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43401 (local address 127.0.98.62:41989)
0924 19:37:26.315504 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.315506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.315748 (+ 242us) server_negotiation.cc:407] Connection header received
0924 19:37:26.316274 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.316275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.316287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.316359 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.317166 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.317936 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.319370 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.319662 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.319751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.321943 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.321944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.321945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.321955 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.323279 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.323692 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.323693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.323694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.323763 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.324011 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.324012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.324013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.324140 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.324199 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.324467 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.324544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:37:26.324699 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.315409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.1:43401)
0924 19:37:26.315573 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.315662 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.316180 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.316452 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.316456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.316804 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.317068 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.317074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.318038 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.318038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.319284 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.319287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.319772 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.319773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.319876 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.320381 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.320390 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.321865 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.323367 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.323374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.323376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.323627 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.323820 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.323821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.323821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.323950 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.324196 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.324200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.324347 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.324422 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.324535 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260924 19:37:26.325558 8158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260924 19:37:26.325637 8158 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:26.325855 8158 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260924 19:37:26.326591 7954 ts_manager.cc:194] Registered new tserver with Master: 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191)
I20260924 19:37:26.327442 7954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43401
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:37:26.331753 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.326337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60815 (local address 127.0.98.62:41989)
0924 19:37:26.326378 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.326380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.326439 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:37:26.326499 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.326500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.326509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.326561 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.326885 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.327581 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.328541 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.328779 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.328835 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.329623 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.329625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.329625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.329631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.330806 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.331095 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.331096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.331096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.331145 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.331304 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.331305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.331305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.331403 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.331453 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.331587 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.331653 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:37:26.331763 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.326271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.1:60815)
0924 19:37:26.326387 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.326397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.326450 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.326563 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.326564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.326723 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.326813 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.326814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.327683 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.327684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.328469 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.328471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.328841 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.328842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.328910 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.329275 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.329280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.329550 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.330876 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.330877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.330877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.331043 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.331203 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.331204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.331204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.331256 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.331452 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.331453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.331503 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.331562 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.331653 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 19:37:26.346252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:26.423954 8167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:26.424154 8167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:26.424175 8167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:26.425349 8167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:26.425386 8167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:26.425406 8167 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:37:26.425422 8167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:26.426863 8167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:26.427040 8167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:26.427211 8167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:37:26.431839 8167 server_base.cc:1034] running on GCE node
W20260924 19:37:26.431779 8173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:26.431829 8172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:26.431784 8175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.432217 8167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:26.432413 8167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:26.433568 8167 hybrid_clock.cc:648] HybridClock initialized: now 1790278646433555 us; error 27 us; skew 500 ppm
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:26.436053 8167 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:37:26.436462 8167 webserver.cc:460] Webserver started at http://127.0.98.2:38835/ using document root <none> and password file <none>
I20260924 19:37:26.436694 8167 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:26.436738 8167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:26.436857 8167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:26.437923 8167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "12ba5cfcd3b14145ab3d82072ec46110"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "ea15cc6b347ac4c3e70a52e77da4b4d0"
server_key_iv: "0bf3d831b52a45f959b061ba579dda74"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.438232 8167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12ba5cfcd3b14145ab3d82072ec46110"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "ea15cc6b347ac4c3e70a52e77da4b4d0"
server_key_iv: "0bf3d831b52a45f959b061ba579dda74"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.439404 8167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:37:26.440141 8182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.440295 8167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.440362 8167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "12ba5cfcd3b14145ab3d82072ec46110"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "ea15cc6b347ac4c3e70a52e77da4b4d0"
server_key_iv: "0bf3d831b52a45f959b061ba579dda74"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.440428 8167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:26.466149 8167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:26.468030 8167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:26.468184 8167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:26.468719 8167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:26.468765 8167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.468791 8167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:26.468819 8167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.476048 8167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35667
I20260924 19:37:26.476119 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35667 every 8 connection(s)
I20260924 19:37:26.476503 8167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:37:26.478543 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.469158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35587 (local address 127.0.98.62:41989)
0924 19:37:26.469220 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.469222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.469443 (+ 221us) server_negotiation.cc:407] Connection header received
0924 19:37:26.470161 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.470163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.470174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.470219 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.471059 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.471801 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.473303 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.473553 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.473607 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.476099 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.476100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.476101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.476109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.477481 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.477851 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.477851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.477852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.477895 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.478067 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.478068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.478068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.478158 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.478200 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.478427 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.478482 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:37:26.478695 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.469129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.2:35587)
0924 19:37:26.469288 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.469339 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.470040 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.470302 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.470306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.470665 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.470952 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.470960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.471897 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.471898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.473202 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.473206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.473634 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.473635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.473786 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.474255 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.474265 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.476022 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.477575 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.477581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.477582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.477792 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.477941 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.477942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.477942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.478027 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.478197 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.478200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.478332 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.478408 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.478518 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 19:37:26.479100 8298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260924 19:37:26.479168 8298 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:26.479331 8298 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260924 19:37:26.479692 7955 ts_manager.cc:194] Registered new tserver with Master: 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667)
I20260924 19:37:26.480183 7955 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35587
I20260924 19:37:26.480611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8167
I20260924 19:37:26.480726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:37:26.480911 392 external_mini_cluster.cc:1428] Setting key c03fe6411e50eee9cd2078cd578e9efa
I20260924 19:37:26.485308 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.480032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53665 (local address 127.0.98.62:41989)
0924 19:37:26.480071 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.480073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.480167 (+ 94us) server_negotiation.cc:407] Connection header received
0924 19:37:26.480230 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.480231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.480241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.480291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.480676 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.481264 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.482114 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.482300 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.482348 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.483140 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.483141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.483142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.483148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.484327 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.484633 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.484634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.484635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.484684 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.484840 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.484841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.484841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.484969 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.485024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.485149 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.485225 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:37:26.485317 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.480002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.2:53665)
0924 19:37:26.480114 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.480124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.480179 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.480334 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.480335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.480509 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.480588 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.480590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.481352 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.481352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.482046 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.482048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.482361 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.482362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.482422 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.482812 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.482817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.483065 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.484395 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.484396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.484397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.484578 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.484738 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.484739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.484739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.484792 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.485022 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.485023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.485063 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.485124 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.485220 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:37:26.502372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:26.581660 8307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:26.581861 8307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:26.581893 8307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:26.583004 8307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:26.583048 8307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:26.583079 8307 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:37:26.583106 8307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:26.584597 8307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:26.584820 8307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:26.585024 8307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:26.589756 8313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.589802 8307 server_base.cc:1034] running on GCE node
W20260924 19:37:26.589808 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:26.589761 8315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:26.590161 8307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:26.590363 8307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:26.591522 8307 hybrid_clock.cc:648] HybridClock initialized: now 1790278646591511 us; error 26 us; skew 500 ppm
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:26.593830 8307 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:37:26.594250 8307 webserver.cc:460] Webserver started at http://127.0.98.3:38655/ using document root <none> and password file <none>
I20260924 19:37:26.594436 8307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:26.594481 8307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:26.594602 8307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:26.595575 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "93f6642568174fb1aa62067513227690"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "719d4f1022277d7dd46a1530c2faf2ce"
server_key_iv: "d866d7397afba4457c58e0a036af2d88"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.595899 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "93f6642568174fb1aa62067513227690"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "719d4f1022277d7dd46a1530c2faf2ce"
server_key_iv: "d866d7397afba4457c58e0a036af2d88"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.597087 8307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:26.597729 8322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.597901 8307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:26.597982 8307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "93f6642568174fb1aa62067513227690"
format_stamp: "Formatted at 2026-09-24 19:37:26 on dist-test-slave-6kf3"
server_key: "719d4f1022277d7dd46a1530c2faf2ce"
server_key_iv: "d866d7397afba4457c58e0a036af2d88"
server_key_version: "encryptionkey@0"
I20260924 19:37:26.598042 8307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:26.609007 8307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:26.610793 8307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:26.610912 8307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:26.611423 8307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:26.611471 8307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.611500 8307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:26.611516 8307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.618029 8307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39757
I20260924 19:37:26.618096 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39757 every 8 connection(s)
I20260924 19:37:26.618366 8307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:37:26.620601 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.611804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42161 (local address 127.0.98.62:41989)
0924 19:37:26.611866 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.611868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.612123 (+ 255us) server_negotiation.cc:407] Connection header received
0924 19:37:26.612778 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.612780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.612793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.612853 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.613672 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.614209 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.615177 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.615355 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.615405 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.617879 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.617881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.617881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.617889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.619454 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.619776 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.619776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.619777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.619830 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.620031 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.620032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.620032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.620154 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.620209 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.620476 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.620540 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:37:26.620666 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.611784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.3:42161)
0924 19:37:26.611975 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.612023 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.612657 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.612943 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.612947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.613290 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.613566 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.613572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.614290 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.614291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.615089 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.615091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.615418 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.615419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.615541 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.615994 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.616009 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.617784 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.619518 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.619522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.619523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.619714 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.619871 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.619872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.619872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.619977 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.620203 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.620206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.620328 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.620400 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.620511 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260924 19:37:26.621099 8438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260924 19:37:26.621174 8438 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:26.621332 8438 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260924 19:37:26.621667 7954 ts_manager.cc:194] Registered new tserver with Master: 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:26.622090 7954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42161
I20260924 19:37:26.626420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8307
I20260924 19:37:26.626519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:37:26.626688 392 external_mini_cluster.cc:1428] Setting key 5bb7653a080d5757fe403f1ae8d0d8e4
I20260924 19:37:26.627182 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.622155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49921 (local address 127.0.98.62:41989)
0924 19:37:26.622195 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.622197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.622339 (+ 142us) server_negotiation.cc:407] Connection header received
0924 19:37:26.622406 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.622407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.622416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.622471 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.622769 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.623233 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.623968 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.624159 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.624215 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.624917 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.624918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.624918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.624925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.626102 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.626458 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.626459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.626460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.626518 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.626717 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.626718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.626719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.626852 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.626914 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.627038 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.627102 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:37:26.627251 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.622066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.3:49921)
0924 19:37:26.622285 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.622296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.622352 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.622474 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.622475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.622644 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.622700 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.622702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.623312 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.623312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.623862 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.623865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.624219 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.624220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.624280 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.624588 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.624592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.624840 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.626182 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.626183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.626183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.626400 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.626605 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.626606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.626606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.626657 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.626916 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.626917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.626960 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.627014 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.627146 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260924 19:37:26.627485 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:37:26.646833 8005 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:26.641427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.0.98.62:41989)
0924 19:37:26.641472 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.641474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.641505 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:37:26.641580 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.641581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.641591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.641634 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.642227 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.642688 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.643463 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.643640 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.643693 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.646770 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49312: BlockingRecv error: recv got EOF from 127.0.0.1:49312 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.654253 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.647870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.98.62:41989)
0924 19:37:26.647908 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.647909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.648012 (+ 103us) server_negotiation.cc:407] Connection header received
0924 19:37:26.648075 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.648076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.648085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.648127 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.648559 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.649013 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.649798 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.650070 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.650133 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.652491 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.652492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.652492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.652498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.653453 (+ 955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.653748 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.653749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.653749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.653795 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.653906 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.653906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.653907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.653987 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.654051 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.654140 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.654187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:37:26.655671 7954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:37:26.657176 7954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:26.662765 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39757 (local address 127.0.0.1:49598)
0924 19:37:26.659507 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.659517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.659567 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.659891 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.659892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.659893 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.659922 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.659923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.660850 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.660850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.661734 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.661736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.662471 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.662472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.662548 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.662563 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.662635 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.662694 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":45,"threads_started":1}
I20260924 19:37:26.662784 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.0.98.3:39757)
0924 19:37:26.659711 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.659716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.659724 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:26.659755 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.659757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.659769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.659833 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.660011 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.660775 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.661807 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.662287 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.662361 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.662441 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.662651 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.662692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":74,"threads_started":1}
I20260924 19:37:26.663554 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.0.1:48258)
0924 19:37:26.659557 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.659566 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.659605 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.659755 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.659757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.659759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.659817 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.659818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.661190 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.661190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.662390 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.662393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.663332 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.663333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.663391 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.663395 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.663447 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.663484 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":60,"thread_start_us":34,"threads_started":1}
I20260924 19:37:26.663600 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.0.98.2:35667)
0924 19:37:26.659491 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.659496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.659508 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:37:26.659627 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.659631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.659647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.659699 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.659977 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.661103 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.662493 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.663177 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.663217 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.663298 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.663477 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.663511 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 19:37:26.663618 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.0.98.1:35191)
0924 19:37:26.659587 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.659591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.659600 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:26.659672 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.659674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.659685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.659745 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.659869 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.661111 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.662479 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.663173 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.663235 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.663298 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.663472 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.663514 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260924 19:37:26.663630 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 (DEFAULT_TABLE table=test-table [id=f7ac7438a61649d4801d898f22a16c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.663996 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8a469875ff4d4fae0ec19f6e017222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.663794 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.659092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35667 (local address 127.0.0.1:39982)
0924 19:37:26.659383 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.659430 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.659492 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.659787 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.659788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.659790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.659840 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.659842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.661189 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.661190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.662393 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.662395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.663349 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.663350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.663399 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.663401 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.663695 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.663736 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":46,"thread_start_us":54,"threads_started":1}
I20260924 19:37:26.664548 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 (DEFAULT_TABLE table=test-table [id=f7ac7438a61649d4801d898f22a16c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.664831 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8a469875ff4d4fae0ec19f6e017222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.666735 8463 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Bootstrap starting.
I20260924 19:37:26.667351 8463 tablet_bootstrap.cc:654] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.667502 8092 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 (DEFAULT_TABLE table=test-table [id=f7ac7438a61649d4801d898f22a16c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.667654 8464 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Bootstrap starting.
I20260924 19:37:26.667658 8463 log.cc:824] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:26.667824 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8a469875ff4d4fae0ec19f6e017222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.668256 8464 tablet_bootstrap.cc:654] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.668427 8463 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: No bootstrap required, opened a new log
I20260924 19:37:26.668491 8463 ts_tablet_manager.cc:1397] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:26.668565 8464 log.cc:824] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:26.669283 8464 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: No bootstrap required, opened a new log
I20260924 19:37:26.669353 8464 ts_tablet_manager.cc:1397] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:37:26.670372 8463 raft_consensus.cc:348] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.670540 8463 raft_consensus.cc:374] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.670572 8463 raft_consensus.cc:729] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f6642568174fb1aa62067513227690, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.670732 8463 consensus_queue.cc:260] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.670979 8463 ts_tablet_manager.cc:1428] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:37:26.671103 8438 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
I20260924 19:37:26.671191 8467 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Bootstrap starting.
I20260924 19:37:26.671249 8464 raft_consensus.cc:348] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.671391 8464 raft_consensus.cc:374] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.671470 8464 raft_consensus.cc:729] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba5cfcd3b14145ab3d82072ec46110, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.671587 8464 consensus_queue.cc:260] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.671793 8467 tablet_bootstrap.cc:654] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.671830 8464 ts_tablet_manager.cc:1428] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:37:26.671918 8298 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
I20260924 19:37:26.672113 8467 log.cc:824] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:26.672778 8467 tablet_bootstrap.cc:492] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: No bootstrap required, opened a new log
I20260924 19:37:26.672834 8467 ts_tablet_manager.cc:1397] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:26.674110 8467 raft_consensus.cc:348] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.674223 8467 raft_consensus.cc:374] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.674252 8467 raft_consensus.cc:729] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.674351 8467 consensus_queue.cc:260] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.674584 8467 ts_tablet_manager.cc:1428] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:37:26.674613 8158 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
W20260924 19:37:26.727066 8299 tablet.cc:2369] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:37:26.805955 8471 raft_consensus.cc:482] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:26.806073 8471 raft_consensus.cc:504] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.806391 8471 leader_election.cc:288] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:26.810207 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.806588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39757 (local address 127.0.98.1:37065)
0924 19:37:26.806786 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.806794 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.806833 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.806999 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.807000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.807001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.807071 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.807073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.807907 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.807908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.809157 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.809160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.809910 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.810007 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.810010 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.810061 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.810100 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":118,"thread_start_us":49,"threads_started":1}
I20260924 19:37:26.810226 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.806685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45999 (local address 127.0.98.2:35667)
0924 19:37:26.806734 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.806736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.806748 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:37:26.806779 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.806780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.806793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.806851 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.807084 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.807835 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809265 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.809866 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809934 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.809943 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.810103 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.810155 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:37:26.810232 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.806839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37065 (local address 127.0.98.3:39757)
0924 19:37:26.806892 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.806894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.806902 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:26.806926 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.806926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.806935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.806975 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.807144 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.807832 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809265 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.809823 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809903 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.809916 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.810103 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.810174 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:37:26.810214 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.806587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35667 (local address 127.0.98.1:45999)
0924 19:37:26.806666 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.806679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.806724 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.806923 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.806924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.806946 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.807023 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.807026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.807912 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.807913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.809157 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.809160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.809932 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.809933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.810007 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.810010 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.810061 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.810100 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":55}
I20260924 19:37:26.810765 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8a469875ff4d4fae0ec19f6e017222" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110" is_pre_election: true
I20260924 19:37:26.810766 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8a469875ff4d4fae0ec19f6e017222" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690" is_pre_election: true
I20260924 19:37:26.810920 8392 raft_consensus.cc:2454] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:26.810920 8252 raft_consensus.cc:2454] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:26.811141 8044 leader_election.cc:302] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ba5cfcd3b14145ab3d82072ec46110, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:26.811287 8471 raft_consensus.cc:2781] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:26.811345 8471 raft_consensus.cc:482] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:26.811360 8471 raft_consensus.cc:3037] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.812058 8471 raft_consensus.cc:504] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.812173 8471 leader_election.cc:288] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Requested vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:26.812345 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8a469875ff4d4fae0ec19f6e017222" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690"
I20260924 19:37:26.812404 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8a469875ff4d4fae0ec19f6e017222" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110"
I20260924 19:37:26.812454 8392 raft_consensus.cc:3037] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.812475 8252 raft_consensus.cc:3037] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.813401 8392 raft_consensus.cc:2454] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:26.813401 8252 raft_consensus.cc:2454] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:26.813846 8043 leader_election.cc:302] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:26.813951 8471 raft_consensus.cc:2781] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:26.814128 8471 raft_consensus.cc:686] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Becoming Leader. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Running, Role: LEADER
I20260924 19:37:26.814224 8471 consensus_queue.cc:237] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } }
I20260924 19:37:26.815165 7955 catalog_manager.cc:5495] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9acd5d4152974c1c869d455539af27eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } health_report { overall_health: HEALTHY } } }
W20260924 19:37:26.822623 8159 tablet.cc:2369] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:37:26.847877 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.845587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.98.1:35191)
0924 19:37:26.845642 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.845644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.845651 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:37:26.845690 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.845691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.845713 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.845778 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:26.845923 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.846394 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.847191 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.847391 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.847444 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.847572 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:26.847679 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:26.847746 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.847781 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.847800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:37:26.849490 8252 raft_consensus.cc:1261] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:26.849490 8392 raft_consensus.cc:1261] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:26.849831 8473 consensus_queue.cc:1035] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:26.849960 8471 consensus_queue.cc:1035] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:26.852960 8477 mvcc.cc:205] Tried to move back new op lower bound from 7332981337492926464 to 7332981337351737344. Current Snapshot: MvccSnapshot[applied={T|T < 7332981337492926464}]
I20260924 19:37:26.854072 7955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:26.854178 7955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:26.854911 7955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Sending DeleteTablet for 3 replicas of tablet 2d8a469875ff4d4fae0ec19f6e017222
I20260924 19:37:26.855288 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260924 19:37:26.855372 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39982
I20260924 19:37:26.855553 8092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8a469875ff4d4fae0ec19f6e017222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260924 19:37:26.855571 8488 tablet_replica.cc:330] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: stopping tablet replica
I20260924 19:37:26.855682 8488 raft_consensus.cc:2229] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:26.855726 8489 tablet_replica.cc:330] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: stopping tablet replica
I20260924 19:37:26.855790 8488 raft_consensus.cc:2258] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:26.855808 8489 raft_consensus.cc:2229] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:37:26.855919 8489 raft_consensus.cc:2258] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:26.856143 7955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:37:26.856236 8488 ts_tablet_manager.cc:1905] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:37:26.856276 7955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:26.856626 8487 tablet_replica.cc:330] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: stopping tablet replica
I20260924 19:37:26.856730 8487 raft_consensus.cc:2229] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:26.856844 8487 raft_consensus.cc:2258] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:26.857842 8489 ts_tablet_manager.cc:1905] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:26.857894 8092 tablet_service.cc:1467] Processing CreateTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (DEFAULT_TABLE table=test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.857954 8488 ts_tablet_manager.cc:1918] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:26.857978 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.858022 8488 log.cc:1196] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/2d8a469875ff4d4fae0ec19f6e017222
I20260924 19:37:26.858271 8488 ts_tablet_manager.cc:1939] T 2d8a469875ff4d4fae0ec19f6e017222 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting consensus metadata
I20260924 19:37:26.859226 8467 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Bootstrap starting.
I20260924 19:37:26.859256 8372 tablet_service.cc:1467] Processing CreateTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (DEFAULT_TABLE table=test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.859349 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.859452 8487 ts_tablet_manager.cc:1905] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:26.859688 8467 tablet_bootstrap.cc:654] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.859227 8489 ts_tablet_manager.cc:1918] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:26.860100 8489 log.cc:1196] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/2d8a469875ff4d4fae0ec19f6e017222
I20260924 19:37:26.860344 8489 ts_tablet_manager.cc:1939] T 2d8a469875ff4d4fae0ec19f6e017222 P 9acd5d4152974c1c869d455539af27eb: Deleting consensus metadata
I20260924 19:37:26.860647 8232 tablet_service.cc:1467] Processing CreateTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (DEFAULT_TABLE table=test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:26.860728 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.860945 8467 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: No bootstrap required, opened a new log
I20260924 19:37:26.860988 8467 ts_tablet_manager.cc:1397] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:26.861150 8487 ts_tablet_manager.cc:1918] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:26.861146 8467 raft_consensus.cc:348] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.861208 8487 log.cc:1196] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/2d8a469875ff4d4fae0ec19f6e017222
I20260924 19:37:26.861214 8467 raft_consensus.cc:374] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.861238 8467 raft_consensus.cc:729] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.861281 8467 consensus_queue.cc:260] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.861379 8467 ts_tablet_manager.cc:1428] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.861626 8487 ts_tablet_manager.cc:1939] T 2d8a469875ff4d4fae0ec19f6e017222 P 93f6642568174fb1aa62067513227690: Deleting consensus metadata
I20260924 19:37:26.861948 8463 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Bootstrap starting.
I20260924 19:37:26.862272 8463 tablet_bootstrap.cc:654] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.862529 8464 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Bootstrap starting.
I20260924 19:37:26.862843 8464 tablet_bootstrap.cc:654] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.863471 7940 catalog_manager.cc:4841] TS 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667): tablet 2d8a469875ff4d4fae0ec19f6e017222 (table test-table [id=f7ac7438a61649d4801d898f22a16c8c]) successfully deleted
I20260924 19:37:26.863600 7939 catalog_manager.cc:4841] TS 93f6642568174fb1aa62067513227690 (127.0.98.3:39757): tablet 2d8a469875ff4d4fae0ec19f6e017222 (table test-table [id=f7ac7438a61649d4801d898f22a16c8c]) successfully deleted
I20260924 19:37:26.863658 7939 catalog_manager.cc:4841] TS 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191): tablet 2d8a469875ff4d4fae0ec19f6e017222 (table test-table [id=f7ac7438a61649d4801d898f22a16c8c]) successfully deleted
I20260924 19:37:26.864004 8464 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: No bootstrap required, opened a new log
I20260924 19:37:26.864049 8464 ts_tablet_manager.cc:1397] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:26.864202 8464 raft_consensus.cc:348] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.864269 8464 raft_consensus.cc:374] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.864295 8464 raft_consensus.cc:729] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba5cfcd3b14145ab3d82072ec46110, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.864333 8464 consensus_queue.cc:260] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.864408 8464 ts_tablet_manager.cc:1428] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.864624 8463 tablet_bootstrap.cc:492] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: No bootstrap required, opened a new log
I20260924 19:37:26.864662 8463 ts_tablet_manager.cc:1397] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:37:26.864787 8463 raft_consensus.cc:348] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.864833 8463 raft_consensus.cc:374] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.864845 8463 raft_consensus.cc:729] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f6642568174fb1aa62067513227690, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.864874 8463 consensus_queue.cc:260] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.864943 8463 ts_tablet_manager.cc:1428] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 19:37:26.868943 8439 tablet.cc:2369] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:37:26.872164 8468 raft_consensus.cc:482] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:26.872242 8468 raft_consensus.cc:504] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.872598 8468 leader_election.cc:288] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191), 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667)
I20260924 19:37:26.876060 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.873042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52713 (local address 127.0.98.2:35667)
0924 19:37:26.873083 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.873085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.873263 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:37:26.873351 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.873352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.873361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.873412 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.873583 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.874147 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875159 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.875692 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875759 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.875771 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.875927 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.875998 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:37:26.876091 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.872756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.98.3:43617)
0924 19:37:26.872916 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.872926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.872973 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.873108 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.873108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.873110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.873174 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.873176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.873969 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.873970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.875119 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.875122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.875840 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.875897 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.875900 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.875979 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.876017 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1}
I20260924 19:37:26.876168 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.872799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35667 (local address 127.0.98.3:52713)
0924 19:37:26.873190 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.873202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.873291 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.873424 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.873425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.873426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:26.873520 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.873522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.874208 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.874210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.875092 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.875094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.875781 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.875852 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.875857 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.875918 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.876012 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260924 19:37:26.876389 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.872851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43617 (local address 127.0.98.1:35191)
0924 19:37:26.872891 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.872893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.872960 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:37:26.873029 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.873029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.873037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.873106 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:26.873221 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.873896 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875216 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.875742 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.875805 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.875813 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.876289 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.876333 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:37:26.876426 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8df59c9a42f4a4693fc4d1f83b8a8ac" candidate_uuid: "93f6642568174fb1aa62067513227690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110" is_pre_election: true
I20260924 19:37:26.876518 8252 raft_consensus.cc:2454] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f6642568174fb1aa62067513227690 in term 0.
I20260924 19:37:26.876675 8324 leader_election.cc:302] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ba5cfcd3b14145ab3d82072ec46110, 93f6642568174fb1aa62067513227690; no voters:
I20260924 19:37:26.876788 8468 raft_consensus.cc:2781] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:26.876820 8468 raft_consensus.cc:482] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:26.876843 8468 raft_consensus.cc:3037] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.876770 8112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8df59c9a42f4a4693fc4d1f83b8a8ac" candidate_uuid: "93f6642568174fb1aa62067513227690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acd5d4152974c1c869d455539af27eb" is_pre_election: true
I20260924 19:37:26.876894 8112 raft_consensus.cc:2454] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f6642568174fb1aa62067513227690 in term 0.
I20260924 19:37:26.877696 8468 raft_consensus.cc:504] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.877835 8468 leader_election.cc:288] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [CANDIDATE]: Term 1 election: Requested vote from peers 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191), 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667)
I20260924 19:37:26.877928 8112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8df59c9a42f4a4693fc4d1f83b8a8ac" candidate_uuid: "93f6642568174fb1aa62067513227690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acd5d4152974c1c869d455539af27eb"
I20260924 19:37:26.877948 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8df59c9a42f4a4693fc4d1f83b8a8ac" candidate_uuid: "93f6642568174fb1aa62067513227690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110"
I20260924 19:37:26.877992 8112 raft_consensus.cc:3037] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.878000 8252 raft_consensus.cc:3037] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:26.878719 8112 raft_consensus.cc:2454] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f6642568174fb1aa62067513227690 in term 1.
I20260924 19:37:26.878728 8252 raft_consensus.cc:2454] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f6642568174fb1aa62067513227690 in term 1.
I20260924 19:37:26.878878 8323 leader_election.cc:302] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:26.878968 8468 raft_consensus.cc:2781] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:26.879087 8468 raft_consensus.cc:686] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 1 LEADER]: Becoming Leader. State: Replica: 93f6642568174fb1aa62067513227690, State: Running, Role: LEADER
I20260924 19:37:26.879148 8468 consensus_queue.cc:237] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.879755 7955 catalog_manager.cc:5495] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f6642568174fb1aa62067513227690 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "93f6642568174fb1aa62067513227690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } health_report { overall_health: HEALTHY } } }
Sep 24 19:37:26 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:26.926585 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.912606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.0.1:49330)
0924 19:37:26.912748 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:26.912760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:26.912821 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:26.912982 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:26.912983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:26.913280 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:26.913330 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.913332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.914213 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.914214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.914731 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:26.914732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.915140 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.915160 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:26.915197 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.922236 (+ 7039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:26.922242 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:26.923758 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:26.925594 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.925602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.925603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.925789 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.925996 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:26.925996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:26.925997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:26.926048 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:26.926324 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:26.926331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:26.926374 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:37:26.926420 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:37:26.926512 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 19:37:26.926708 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:26.912766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.98.62:41989)
0924 19:37:26.912830 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:26.912832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:26.912873 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:37:26.912902 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:26.912903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:26.912916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:26.912979 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:26.913407 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.914138 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.914799 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:26.915070 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:26.915149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:26.923867 (+ 8718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:26.923869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:26.923870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:26.923879 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:26.925510 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.925862 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.925863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.925863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.925931 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:26.926103 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:26.926104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:26.926105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:26.926254 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:26.926326 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:26.926467 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:37:26.926578 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":38}
I20260924 19:37:26.928712 7955 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49330:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:37:26.928963 7955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:26.930306 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 59fc240887994d8dbd962c6341177498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=650a5cdd65ea47a3bb2294c7f6e69ffa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:37:26.930336 8092 tablet_service.cc:1467] Processing CreateTablet for tablet 59fc240887994d8dbd962c6341177498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=650a5cdd65ea47a3bb2294c7f6e69ffa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:37:26.930382 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 59fc240887994d8dbd962c6341177498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=650a5cdd65ea47a3bb2294c7f6e69ffa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:37:26.930450 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fc240887994d8dbd962c6341177498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.930450 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fc240887994d8dbd962c6341177498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.930523 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fc240887994d8dbd962c6341177498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:26.931807 8463 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: Bootstrap starting.
I20260924 19:37:26.932047 8464 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: Bootstrap starting.
I20260924 19:37:26.932080 8467 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: Bootstrap starting.
I20260924 19:37:26.932230 8463 tablet_bootstrap.cc:654] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.932444 8464 tablet_bootstrap.cc:654] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.932519 8467 tablet_bootstrap.cc:654] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:26.933010 8463 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: No bootstrap required, opened a new log
I20260924 19:37:26.933056 8463 ts_tablet_manager.cc:1397] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:26.933209 8464 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: No bootstrap required, opened a new log
I20260924 19:37:26.933250 8464 ts_tablet_manager.cc:1397] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:26.933308 8467 tablet_bootstrap.cc:492] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: No bootstrap required, opened a new log
I20260924 19:37:26.933349 8467 ts_tablet_manager.cc:1397] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:26.933327 8463 raft_consensus.cc:348] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933398 8463 raft_consensus.cc:374] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.933377 8464 raft_consensus.cc:348] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933425 8463 raft_consensus.cc:729] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f6642568174fb1aa62067513227690, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.933431 8464 raft_consensus.cc:374] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.933449 8464 raft_consensus.cc:729] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba5cfcd3b14145ab3d82072ec46110, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.933477 8463 consensus_queue.cc:260] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933482 8464 consensus_queue.cc:260] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933477 8467 raft_consensus.cc:348] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933543 8467 raft_consensus.cc:374] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:26.933563 8467 raft_consensus.cc:729] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Initialized, Role: FOLLOWER
I20260924 19:37:26.933559 8464 ts_tablet_manager.cc:1428] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.933559 8463 ts_tablet_manager.cc:1428] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.933606 8467 consensus_queue.cc:260] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:26.933688 8467 ts_tablet_manager.cc:1428] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:26.933696 8469 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:26.933765 8469 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:26.933771 8468 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:26.933828 8468 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:26.933804 8483 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:26.933867 8483 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:27.213899 8483 raft_consensus.cc:482] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:27.214003 8483 raft_consensus.cc:504] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.214159 8483 leader_election.cc:288] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.214406 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fc240887994d8dbd962c6341177498" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690" is_pre_election: true
I20260924 19:37:27.214398 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fc240887994d8dbd962c6341177498" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110" is_pre_election: true
I20260924 19:37:27.214500 8392 raft_consensus.cc:2454] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.214529 8252 raft_consensus.cc:2454] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.214650 8043 leader_election.cc:302] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.214747 8483 raft_consensus.cc:2781] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:27.214782 8483 raft_consensus.cc:482] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:27.214805 8483 raft_consensus.cc:3037] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.215539 8483 raft_consensus.cc:504] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.215653 8483 leader_election.cc:288] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Requested vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.215843 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fc240887994d8dbd962c6341177498" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110"
I20260924 19:37:27.215847 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fc240887994d8dbd962c6341177498" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690"
I20260924 19:37:27.215953 8392 raft_consensus.cc:3037] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.215953 8252 raft_consensus.cc:3037] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.216697 8252 raft_consensus.cc:2454] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.216697 8392 raft_consensus.cc:2454] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.216876 8043 leader_election.cc:302] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.216980 8483 raft_consensus.cc:2781] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:27.217041 8483 raft_consensus.cc:686] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Becoming Leader. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Running, Role: LEADER
I20260924 19:37:27.217094 8483 consensus_queue.cc:237] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.217372 8473 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: TxnStatusTablet state changed. Reason: New leader 9acd5d4152974c1c869d455539af27eb. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.217468 8473 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:37:27.217686 8506 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:37:27.217665 7954 catalog_manager.cc:5495] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9acd5d4152974c1c869d455539af27eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260924 19:37:27.217816 8506 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:37:27.289219 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.286579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.0.1:48276)
0924 19:37:27.286650 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:27.286662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:27.286708 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:27.286877 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:27.286877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:27.286878 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:37:27.286927 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.286928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.287807 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.287807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.288410 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.288412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.288870 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.288871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.288912 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.288928 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.289090 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.289095 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:37:27.289123 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 19:37:27.289160 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:37:27.289234 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.286671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.98.1:35191)
0924 19:37:27.286745 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.286747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.286757 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:37:27.286783 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.286784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.286797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.286871 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.286989 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.287731 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.288498 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.288800 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.288876 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.289000 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.289037 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.289088 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.289133 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.289154 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:37:27.290302 8392 raft_consensus.cc:1261] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.290316 8252 raft_consensus.cc:1261] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.290540 8473 consensus_queue.cc:1035] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.290659 8483 consensus_queue.cc:1035] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.291041 8468 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: TxnStatusTablet state changed. Reason: New leader 9acd5d4152974c1c869d455539af27eb. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.291126 8468 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:27.291335 8469 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: TxnStatusTablet state changed. Reason: New leader 9acd5d4152974c1c869d455539af27eb. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.291419 8469 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:27.291664 8483 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.291746 8483 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:37:27.291841 8476 mvcc.cc:205] Tried to move back new op lower bound from 7332981339299368960 to 7332981339001552896. Current Snapshot: MvccSnapshot[applied={T|T < 7332981339299368960 or (T in {7332981339299368960})}]
I20260924 19:37:27.292438 8469 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.292522 8469 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 12ba5cfcd3b14145ab3d82072ec46110: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:27.292893 8473 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.292977 8473 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 9acd5d4152974c1c869d455539af27eb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:37:27.293435 8478 mvcc.cc:205] Tried to move back new op lower bound from 7332981339299368960 to 7332981339001552896. Current Snapshot: MvccSnapshot[applied={T|T < 7332981339299368960}]
I20260924 19:37:27.293547 8468 tablet_replica.cc:439] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } } }
I20260924 19:37:27.293610 8468 tablet_replica.cc:442] T 59fc240887994d8dbd962c6341177498 P 93f6642568174fb1aa62067513227690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:37:27.295543 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.293193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.0.98.3:39757)
0924 19:37:27.293227 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.293229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.293252 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:37:27.293285 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.293286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.293305 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.293365 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.293534 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.294142 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.294872 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.295064 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.295117 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.295244 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.295347 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.295405 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.295449 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.295467 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:37:27.299881 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.297785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41807 (local address 127.0.98.1:35191)
0924 19:37:27.297828 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.297830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.297952 (+ 122us) server_negotiation.cc:407] Connection header received
0924 19:37:27.297997 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.297998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.298010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.298048 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.298162 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.298593 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.299292 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.299476 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.299525 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.299671 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.299702 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.299742 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.299792 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.299811 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:37:27.299907 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.297742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.98.3:41807)
0924 19:37:27.297892 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:27.297903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:27.297953 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:27.298057 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:27.298058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:27.298059 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:37:27.298113 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.298114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.298660 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.298661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.299225 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.299227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.299533 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.299534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.299595 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.299608 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.299750 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.299752 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:37:27.299790 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 19:37:27.299829 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260924 19:37:27.302079 8252 raft_consensus.cc:1261] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Refusing update from remote peer 93f6642568174fb1aa62067513227690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.302074 8112 raft_consensus.cc:1261] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Refusing update from remote peer 93f6642568174fb1aa62067513227690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.302320 8494 consensus_queue.cc:1035] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.302443 8468 consensus_queue.cc:1035] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.309777 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.307274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39757 (local address 127.0.98.1:51177)
0924 19:37:27.307446 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:27.307455 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:27.307506 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:27.307617 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:27.307618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:27.307620 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:37:27.307664 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.307665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.308287 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.308288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.308938 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.308941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.309417 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.309417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.309461 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.309476 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.309652 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.309654 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:37:27.309682 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 19:37:27.309716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260924 19:37:27.309775 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.307331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51177 (local address 127.0.98.3:39757)
0924 19:37:27.307366 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.307368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.307508 (+ 140us) server_negotiation.cc:407] Connection header received
0924 19:37:27.307556 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.307557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.307569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.307606 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.307715 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.308221 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.309067 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.309346 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.309419 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.309569 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.309602 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.309653 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.309687 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.309708 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:37:27.315654 7954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.315745 7954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.316357 7954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Sending DeleteTablet for 3 replicas of tablet b8df59c9a42f4a4693fc4d1f83b8a8ac
I20260924 19:37:27.316576 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39982
I20260924 19:37:27.316642 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260924 19:37:27.316676 8488 tablet_replica.cc:330] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: stopping tablet replica
I20260924 19:37:27.316726 8487 tablet_replica.cc:330] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: stopping tablet replica
I20260924 19:37:27.316726 8488 raft_consensus.cc:2229] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.316768 8488 raft_consensus.cc:2258] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.316783 8487 raft_consensus.cc:2229] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:37:27.316854 8487 raft_consensus.cc:2258] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.317091 8488 ts_tablet_manager.cc:1905] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.317157 8092 tablet_service.cc:1514] Processing DeleteTablet for tablet b8df59c9a42f4a4693fc4d1f83b8a8ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260924 19:37:27.317229 8489 tablet_replica.cc:330] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: stopping tablet replica
I20260924 19:37:27.317268 8489 raft_consensus.cc:2229] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.317298 8489 raft_consensus.cc:2258] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.317601 8489 ts_tablet_manager.cc:1905] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.317641 8487 ts_tablet_manager.cc:1905] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.318732 8488 ts_tablet_manager.cc:1918] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:37:27.318778 8488 log.cc:1196] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/b8df59c9a42f4a4693fc4d1f83b8a8ac
I20260924 19:37:27.318971 8488 ts_tablet_manager.cc:1939] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 12ba5cfcd3b14145ab3d82072ec46110: Deleting consensus metadata
I20260924 19:37:27.318996 8489 ts_tablet_manager.cc:1918] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:37:27.319034 8489 log.cc:1196] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/b8df59c9a42f4a4693fc4d1f83b8a8ac
I20260924 19:37:27.319205 8489 ts_tablet_manager.cc:1939] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 9acd5d4152974c1c869d455539af27eb: Deleting consensus metadata
I20260924 19:37:27.319207 8487 ts_tablet_manager.cc:1918] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:37:27.319265 8487 log.cc:1196] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/b8df59c9a42f4a4693fc4d1f83b8a8ac
I20260924 19:37:27.319244 7940 catalog_manager.cc:4841] TS 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667): tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (table test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]) successfully deleted
I20260924 19:37:27.319451 8487 ts_tablet_manager.cc:1939] T b8df59c9a42f4a4693fc4d1f83b8a8ac P 93f6642568174fb1aa62067513227690: Deleting consensus metadata
I20260924 19:37:27.319445 7939 catalog_manager.cc:4841] TS 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191): tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (table test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]) successfully deleted
I20260924 19:37:27.325465 7939 catalog_manager.cc:4841] TS 93f6642568174fb1aa62067513227690 (127.0.98.3:39757): tablet b8df59c9a42f4a4693fc4d1f83b8a8ac (table test-table [id=82ef78bc632d4e5dbb42684520fb6ee9]) successfully deleted
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[7910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:37:27.346426 8005 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:27.340512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.0.98.62:41989)
0924 19:37:27.340558 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.340560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.340698 (+ 138us) server_negotiation.cc:407] Connection header received
0924 19:37:27.340755 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.340756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.340777 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.340837 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:27.341466 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.342275 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.343251 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.343552 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.343615 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.346358 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49342: BlockingRecv error: recv got EOF from 127.0.0.1:49342 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[7910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:37:27.353829 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.347106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.0.98.62:41989)
0924 19:37:27.347147 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.347149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.347174 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:37:27.347223 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.347224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.347233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.347295 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:27.347678 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.348464 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.349082 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.349359 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.349425 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.351441 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:27.351443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:27.351443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:27.351450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:27.352902 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:27.353177 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:27.353179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:27.353179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:27.353233 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:27.353373 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:27.353374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:27.353374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:27.353495 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:27.353575 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.353654 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.353730 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:37:27.355177 7954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:37:27.355342 7954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:27.356599 8372 tablet_service.cc:1467] Processing CreateTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 (DEFAULT_TABLE table=test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.356648 8092 tablet_service.cc:1467] Processing CreateTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 (DEFAULT_TABLE table=test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.356649 8232 tablet_service.cc:1467] Processing CreateTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 (DEFAULT_TABLE table=test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.356724 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7402447cadf4933a31f4ddeb296bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.356765 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7402447cadf4933a31f4ddeb296bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.356765 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7402447cadf4933a31f4ddeb296bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.357944 8463 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Bootstrap starting.
I20260924 19:37:27.358043 8467 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Bootstrap starting.
I20260924 19:37:27.358043 8464 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Bootstrap starting.
I20260924 19:37:27.358319 8463 tablet_bootstrap.cc:654] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.358343 8467 tablet_bootstrap.cc:654] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.358400 8464 tablet_bootstrap.cc:654] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.359022 8463 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: No bootstrap required, opened a new log
I20260924 19:37:27.359067 8463 ts_tablet_manager.cc:1397] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:27.359154 8467 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: No bootstrap required, opened a new log
I20260924 19:37:27.359194 8467 ts_tablet_manager.cc:1397] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:27.359201 8463 raft_consensus.cc:348] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359261 8463 raft_consensus.cc:374] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.359278 8463 raft_consensus.cc:729] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f6642568174fb1aa62067513227690, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.359292 8464 tablet_bootstrap.cc:492] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: No bootstrap required, opened a new log
I20260924 19:37:27.359287 8467 raft_consensus.cc:348] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359326 8464 ts_tablet_manager.cc:1397] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:37:27.359315 8463 consensus_queue.cc:260] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359349 8467 raft_consensus.cc:374] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.359368 8467 raft_consensus.cc:729] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.359401 8463 ts_tablet_manager.cc:1428] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:27.359411 8467 consensus_queue.cc:260] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359485 8467 ts_tablet_manager.cc:1428] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:27.359462 8464 raft_consensus.cc:348] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359516 8464 raft_consensus.cc:374] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.359534 8464 raft_consensus.cc:729] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba5cfcd3b14145ab3d82072ec46110, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.359570 8464 consensus_queue.cc:260] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.359644 8464 ts_tablet_manager.cc:1428] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:27.379347 8484 raft_consensus.cc:482] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:27.379432 8484 raft_consensus.cc:504] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.379588 8484 leader_election.cc:288] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.379742 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7402447cadf4933a31f4ddeb296bbf4" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690" is_pre_election: true
I20260924 19:37:27.379832 8392 raft_consensus.cc:2454] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.379784 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7402447cadf4933a31f4ddeb296bbf4" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110" is_pre_election: true
I20260924 19:37:27.379890 8252 raft_consensus.cc:2454] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.380002 8043 leader_election.cc:302] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.380111 8484 raft_consensus.cc:2781] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:27.380146 8484 raft_consensus.cc:482] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:27.380164 8484 raft_consensus.cc:3037] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.380903 8484 raft_consensus.cc:504] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.381019 8484 leader_election.cc:288] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Requested vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.381137 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7402447cadf4933a31f4ddeb296bbf4" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110"
I20260924 19:37:27.381213 8252 raft_consensus.cc:3037] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.381187 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7402447cadf4933a31f4ddeb296bbf4" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690"
I20260924 19:37:27.381245 8392 raft_consensus.cc:3037] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.381739 8392 raft_consensus.cc:2454] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.381878 8043 leader_election.cc:302] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f6642568174fb1aa62067513227690, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.381953 8484 raft_consensus.cc:2781] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:27.381965 8252 raft_consensus.cc:2454] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.381992 8484 raft_consensus.cc:686] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Becoming Leader. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Running, Role: LEADER
I20260924 19:37:27.382038 8484 consensus_queue.cc:237] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.382506 7954 catalog_manager.cc:5495] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9acd5d4152974c1c869d455539af27eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260924 19:37:27.390370 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.388110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.98.1:35191)
0924 19:37:27.388152 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.388154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.388162 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:27.388207 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.388208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.388218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.388264 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.388348 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.389030 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.389714 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.390017 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.390083 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.390170 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.390201 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.390251 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.390284 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.390303 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:37:27.391000 8252 raft_consensus.cc:1261] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.391031 8392 raft_consensus.cc:1261] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.391191 8484 consensus_queue.cc:1035] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.391286 8471 consensus_queue.cc:1035] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.393857 7954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.393930 7954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.394488 7954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Sending DeleteTablet for 3 replicas of tablet f7402447cadf4933a31f4ddeb296bbf4
I20260924 19:37:27.394701 8092 tablet_service.cc:1514] Processing DeleteTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260924 19:37:27.394728 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260924 19:37:27.394816 8487 tablet_replica.cc:330] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: stopping tablet replica
I20260924 19:37:27.394809 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet f7402447cadf4933a31f4ddeb296bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39982
I20260924 19:37:27.394860 8487 raft_consensus.cc:2229] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.394886 8489 tablet_replica.cc:330] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: stopping tablet replica
I20260924 19:37:27.394896 8487 raft_consensus.cc:2258] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.394891 8488 tablet_replica.cc:330] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: stopping tablet replica
I20260924 19:37:27.394933 8488 raft_consensus.cc:2229] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.394933 8489 raft_consensus.cc:2229] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:37:27.394980 8488 raft_consensus.cc:2258] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.394980 8489 raft_consensus.cc:2258] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.395180 8487 ts_tablet_manager.cc:1905] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.395237 8488 ts_tablet_manager.cc:1905] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.395623 8489 ts_tablet_manager.cc:1905] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.396658 8487 ts_tablet_manager.cc:1918] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:27.396713 8487 log.cc:1196] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/f7402447cadf4933a31f4ddeb296bbf4
I20260924 19:37:27.396652 7954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:37:27.396778 7954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:27.396941 8487 ts_tablet_manager.cc:1939] T f7402447cadf4933a31f4ddeb296bbf4 P 93f6642568174fb1aa62067513227690: Deleting consensus metadata
I20260924 19:37:27.397078 8489 ts_tablet_manager.cc:1918] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:27.397121 8489 log.cc:1196] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/f7402447cadf4933a31f4ddeb296bbf4
I20260924 19:37:27.397229 7939 catalog_manager.cc:4841] TS 93f6642568174fb1aa62067513227690 (127.0.98.3:39757): tablet f7402447cadf4933a31f4ddeb296bbf4 (table test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]) successfully deleted
I20260924 19:37:27.397302 8489 ts_tablet_manager.cc:1939] T f7402447cadf4933a31f4ddeb296bbf4 P 9acd5d4152974c1c869d455539af27eb: Deleting consensus metadata
I20260924 19:37:27.397615 8488 ts_tablet_manager.cc:1918] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:27.397680 8488 log.cc:1196] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/f7402447cadf4933a31f4ddeb296bbf4
I20260924 19:37:27.397889 8488 ts_tablet_manager.cc:1939] T f7402447cadf4933a31f4ddeb296bbf4 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting consensus metadata
I20260924 19:37:27.397935 8092 tablet_service.cc:1467] Processing CreateTablet for tablet bfa55bc7e8624802ac1b4be63baada64 (DEFAULT_TABLE table=test-table [id=321a705733124f5a853b7c1a86416eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.397935 8232 tablet_service.cc:1467] Processing CreateTablet for tablet bfa55bc7e8624802ac1b4be63baada64 (DEFAULT_TABLE table=test-table [id=321a705733124f5a853b7c1a86416eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.398026 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa55bc7e8624802ac1b4be63baada64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.398026 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa55bc7e8624802ac1b4be63baada64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.398178 8372 tablet_service.cc:1467] Processing CreateTablet for tablet bfa55bc7e8624802ac1b4be63baada64 (DEFAULT_TABLE table=test-table [id=321a705733124f5a853b7c1a86416eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:27.398267 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa55bc7e8624802ac1b4be63baada64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:27.399070 8464 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Bootstrap starting.
I20260924 19:37:27.399072 8467 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: Bootstrap starting.
I20260924 19:37:27.399333 8463 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Bootstrap starting.
I20260924 19:37:27.399340 8467 tablet_bootstrap.cc:654] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.399432 8464 tablet_bootstrap.cc:654] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.399650 8463 tablet_bootstrap.cc:654] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:27.400051 7940 catalog_manager.cc:4841] TS 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667): tablet f7402447cadf4933a31f4ddeb296bbf4 (table test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]) successfully deleted
I20260924 19:37:27.400245 7939 catalog_manager.cc:4841] TS 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191): tablet f7402447cadf4933a31f4ddeb296bbf4 (table test-table [id=8b4a5cf93361419b94e554c05ec0d2ef]) successfully deleted
I20260924 19:37:27.400382 8463 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: No bootstrap required, opened a new log
I20260924 19:37:27.400431 8463 ts_tablet_manager.cc:1397] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:27.400595 8463 raft_consensus.cc:348] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.400631 8464 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: No bootstrap required, opened a new log
I20260924 19:37:27.400668 8463 raft_consensus.cc:374] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.400678 8464 ts_tablet_manager.cc:1397] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:37:27.400689 8463 raft_consensus.cc:729] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f6642568174fb1aa62067513227690, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.400727 8463 consensus_queue.cc:260] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.400825 8463 ts_tablet_manager.cc:1428] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:27.400820 8464 raft_consensus.cc:348] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.400888 8464 raft_consensus.cc:374] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.400911 8464 raft_consensus.cc:729] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba5cfcd3b14145ab3d82072ec46110, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.400954 8464 consensus_queue.cc:260] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.401047 8464 ts_tablet_manager.cc:1428] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:27.401233 8467 tablet_bootstrap.cc:492] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: No bootstrap required, opened a new log
I20260924 19:37:27.401278 8467 ts_tablet_manager.cc:1397] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 19:37:27.401377 8467 raft_consensus.cc:348] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.401434 8467 raft_consensus.cc:374] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:27.401446 8467 raft_consensus.cc:729] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Initialized, Role: FOLLOWER
I20260924 19:37:27.401474 8467 consensus_queue.cc:260] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.401552 8467 ts_tablet_manager.cc:1428] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:27.508360 8484 raft_consensus.cc:482] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:27.508456 8484 raft_consensus.cc:504] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.508606 8484 leader_election.cc:288] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.508795 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa55bc7e8624802ac1b4be63baada64" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110" is_pre_election: true
I20260924 19:37:27.508831 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa55bc7e8624802ac1b4be63baada64" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690" is_pre_election: true
I20260924 19:37:27.508894 8252 raft_consensus.cc:2454] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.508903 8392 raft_consensus.cc:2454] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 0.
I20260924 19:37:27.509044 8044 leader_election.cc:302] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ba5cfcd3b14145ab3d82072ec46110, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.509130 8484 raft_consensus.cc:2781] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:27.509178 8484 raft_consensus.cc:482] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:27.509192 8484 raft_consensus.cc:3037] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.509773 8484 raft_consensus.cc:504] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.509876 8484 leader_election.cc:288] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Requested vote from peers 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667), 93f6642568174fb1aa62067513227690 (127.0.98.3:39757)
I20260924 19:37:27.510021 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa55bc7e8624802ac1b4be63baada64" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ba5cfcd3b14145ab3d82072ec46110"
I20260924 19:37:27.510020 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa55bc7e8624802ac1b4be63baada64" candidate_uuid: "9acd5d4152974c1c869d455539af27eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f6642568174fb1aa62067513227690"
I20260924 19:37:27.510109 8252 raft_consensus.cc:3037] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.510109 8392 raft_consensus.cc:3037] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:27.510829 8392 raft_consensus.cc:2454] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.510830 8252 raft_consensus.cc:2454] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9acd5d4152974c1c869d455539af27eb in term 1.
I20260924 19:37:27.511008 8044 leader_election.cc:302] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ba5cfcd3b14145ab3d82072ec46110, 9acd5d4152974c1c869d455539af27eb; no voters:
I20260924 19:37:27.511119 8484 raft_consensus.cc:2781] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:27.511183 8484 raft_consensus.cc:686] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Becoming Leader. State: Replica: 9acd5d4152974c1c869d455539af27eb, State: Running, Role: LEADER
I20260924 19:37:27.511231 8484 consensus_queue.cc:237] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } }
I20260924 19:37:27.511740 7955 catalog_manager.cc:5495] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9acd5d4152974c1c869d455539af27eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9acd5d4152974c1c869d455539af27eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acd5d4152974c1c869d455539af27eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260924 19:37:27.521355 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.519379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47981 (local address 127.0.98.1:35191)
0924 19:37:27.519421 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:27.519423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:27.519430 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:37:27.519476 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:27.519477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:27.519485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:27.519526 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:27.519626 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.520130 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.520820 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.520989 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.521039 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.521153 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.521183 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.521220 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:27.521265 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:37:27.521283 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260924 19:37:27.521467 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.519328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.98.1:47981)
0924 19:37:27.519377 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:27.519387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:27.519426 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:27.519526 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:27.519527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:27.519528 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:37:27.519576 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.519577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.520187 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.520188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.520755 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:27.520757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:27.521060 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:27.521060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:27.521099 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:27.521105 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:37:27.521224 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:37:27.521226 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:37:27.521282 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:37:27.521323 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":2}
I20260924 19:37:27.523124 8392 raft_consensus.cc:1261] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.523146 8252 raft_consensus.cc:1261] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Refusing update from remote peer 9acd5d4152974c1c869d455539af27eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:27.523314 8473 consensus_queue.cc:1035] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ba5cfcd3b14145ab3d82072ec46110" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.523419 8483 consensus_queue.cc:1035] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f6642568174fb1aa62067513227690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:27.532179 7953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.532267 7953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:27.532776 7953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1351a10d306d4c26ad4e8a5a6b0506de: Sending DeleteTablet for 3 replicas of tablet bfa55bc7e8624802ac1b4be63baada64
I20260924 19:37:27.532954 8092 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa55bc7e8624802ac1b4be63baada64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260924 19:37:27.532990 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa55bc7e8624802ac1b4be63baada64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260924 19:37:27.533004 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa55bc7e8624802ac1b4be63baada64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39982
I20260924 19:37:27.533046 8489 tablet_replica.cc:330] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: stopping tablet replica
I20260924 19:37:27.533072 8487 tablet_replica.cc:330] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: stopping tablet replica
I20260924 19:37:27.533074 8488 tablet_replica.cc:330] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: stopping tablet replica
I20260924 19:37:27.533100 8489 raft_consensus.cc:2229] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:37:27.533119 8488 raft_consensus.cc:2229] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.533120 8487 raft_consensus.cc:2229] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:27.533154 8488 raft_consensus.cc:2258] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.533154 8487 raft_consensus.cc:2258] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.533154 8489 raft_consensus.cc:2258] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:27.533437 8488 ts_tablet_manager.cc:1905] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.533437 8489 ts_tablet_manager.cc:1905] T bfa55bc7e8624802ac1b4be63baada64 P 9acd5d4152974c1c869d455539af27eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.533655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8027
I20260924 19:37:27.534224 8487 ts_tablet_manager.cc:1905] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:27.534914 8488 ts_tablet_manager.cc:1918] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:37:27.534973 8488 log.cc:1196] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/bfa55bc7e8624802ac1b4be63baada64
I20260924 19:37:27.535154 8488 ts_tablet_manager.cc:1939] T bfa55bc7e8624802ac1b4be63baada64 P 12ba5cfcd3b14145ab3d82072ec46110: Deleting consensus metadata
I20260924 19:37:27.535543 8487 ts_tablet_manager.cc:1918] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:37:27.535591 8487 log.cc:1196] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/bfa55bc7e8624802ac1b4be63baada64
I20260924 19:37:27.535712 8487 ts_tablet_manager.cc:1939] T bfa55bc7e8624802ac1b4be63baada64 P 93f6642568174fb1aa62067513227690: Deleting consensus metadata
I20260924 19:37:27.535979 7940 catalog_manager.cc:4841] TS 12ba5cfcd3b14145ab3d82072ec46110 (127.0.98.2:35667): tablet bfa55bc7e8624802ac1b4be63baada64 (table test-table [id=321a705733124f5a853b7c1a86416eec]) successfully deleted
I20260924 19:37:27.536108 7939 catalog_manager.cc:4841] TS 93f6642568174fb1aa62067513227690 (127.0.98.3:39757): tablet bfa55bc7e8624802ac1b4be63baada64 (table test-table [id=321a705733124f5a853b7c1a86416eec]) successfully deleted
W20260924 19:37:27.539779 7939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35191 (error 108)
I20260924 19:37:27.539970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8167
I20260924 19:37:27.540597 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:27.540089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35191 (local address 127.0.0.1:48284)
0924 19:37:27.540522 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:27.540560 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":35,"threads_started":1}
W20260924 19:37:27.540683 7939 catalog_manager.cc:4584] TS 9acd5d4152974c1c869d455539af27eb (127.0.98.1:35191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfa55bc7e8624802ac1b4be63baada64: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35191: connect: Connection refused (error 111)
I20260924 19:37:27.544806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8307
I20260924 19:37:27.549124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 7923
2026-09-24T19:37:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2586 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[8551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[8551](info): set up 2 sockets
Sep 24 19:37:27 dist-test-slave-6kf3 krb5kdc[8551](info): commencing operation
krb5kdc: starting...
W20260924 19:37:28.495913 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:37:28 dist-test-slave-6kf3 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:37:28.601824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:28.684162 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:28.684376 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:28.684408 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:28.685550 8564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:28.685597 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:28.685629 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:28.685652 8564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:28.685678 8564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:28.687211 8564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:28.687457 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:28.687675 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:28.692260 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:28.692276 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:28.692294 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:28.692405 8564 server_base.cc:1034] running on GCE node
I20260924 19:37:28.692762 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:28.693032 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:28.694195 8564 hybrid_clock.cc:648] HybridClock initialized: now 1790278648694178 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:37:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1150 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:37:28 dist-test-slave-6kf3 krb5kdc[8577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:37:28 dist-test-slave-6kf3 krb5kdc[8577](info): set up 2 sockets
Sep 24 19:37:28 dist-test-slave-6kf3 krb5kdc[8577](info): commencing operation
krb5kdc: starting...
W20260924 19:37:29.644008 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:37:29 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:37:29.752394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:29.831497 8590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:29.831743 8590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:29.831769 8590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:29.833001 8590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:29.833055 8590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:29.833070 8590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:29.833083 8590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:29.833096 8590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:29.834824 8590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:29.835085 8590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:29.835333 8590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:29.840186 8595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:29.840196 8596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:29.840248 8598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:29.840579 8590 server_base.cc:1034] running on GCE node
I20260924 19:37:29.840858 8590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:29.841122 8590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:29.842301 8590 hybrid_clock.cc:648] HybridClock initialized: now 1790278649842283 us; error 32 us; skew 500 ppm
Sep 24 19:37:29 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:29.844949 8590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:37:29.845376 8590 webserver.cc:460] Webserver started at http://127.0.98.62:32787/ using document root <none> and password file <none>
I20260924 19:37:29.845595 8590 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:29.845642 8590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:29.845772 8590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:29.846822 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "6d34aec2c68c47fba54f61c04da7f2af"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "98302deff0b81aecfb597785ed093585"
server_key_iv: "068046bc5ae4a670d9b66f5881bf25df"
server_key_version: "encryptionkey@0"
I20260924 19:37:29.847148 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d34aec2c68c47fba54f61c04da7f2af"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "98302deff0b81aecfb597785ed093585"
server_key_iv: "068046bc5ae4a670d9b66f5881bf25df"
server_key_version: "encryptionkey@0"
I20260924 19:37:29.848508 8590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:29.849282 8605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:29.849445 8590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:29.849535 8590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "6d34aec2c68c47fba54f61c04da7f2af"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "98302deff0b81aecfb597785ed093585"
server_key_iv: "068046bc5ae4a670d9b66f5881bf25df"
server_key_version: "encryptionkey@0"
I20260924 19:37:29.849601 8590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:29.859457 8590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:29.861274 8590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:29.861420 8590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:29.865401 8590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42509
I20260924 19:37:29.865428 8657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42509 every 8 connection(s)
I20260924 19:37:29.865916 8590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:29.866530 8658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:29.866681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8590
I20260924 19:37:29.866777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:37:29.866979 392 external_mini_cluster.cc:1428] Setting key b21a07c5da9230c6d1735dafc7231faf
I20260924 19:37:29.869685 8658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Bootstrap starting.
I20260924 19:37:29.870298 8658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:29.870591 8658 log.cc:824] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:29.871234 8658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: No bootstrap required, opened a new log
Sep 24 19:37:29 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:29.873515 8658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:29.873643 8658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:29.873672 8658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34aec2c68c47fba54f61c04da7f2af, State: Initialized, Role: FOLLOWER
I20260924 19:37:29.873806 8658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:29.873880 8658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:29.873911 8658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:29.873957 8658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:29.874588 8658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:29.874704 8658 leader_election.cc:302] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d34aec2c68c47fba54f61c04da7f2af; no voters:
I20260924 19:37:29.874893 8658 leader_election.cc:288] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:37:29.875018 8663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:29.875145 8658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:29.875160 8663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 LEADER]: Becoming Leader. State: Replica: 6d34aec2c68c47fba54f61c04da7f2af, State: Running, Role: LEADER
I20260924 19:37:29.875273 8663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:29.875559 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d34aec2c68c47fba54f61c04da7f2af. Latest consensus state: current_term: 1 leader_uuid: "6d34aec2c68c47fba54f61c04da7f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 19:37:29.875626 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:29.875785 8664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d34aec2c68c47fba54f61c04da7f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 19:37:29.875835 8664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: This master's current role is: LEADER
W20260924 19:37:29.876936 8679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:37:29.876986 8679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:37:29.877038 8671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:29.877199 8671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:29.878125 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:29.867507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.0.98.62:42509)
0924 19:37:29.867660 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:29.867668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:29.867699 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:37:29.868227 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:29.868234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:29.868284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:29.868397 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:29.869122 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:29.870054 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:29.871018 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:29.871483 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:29.871577 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:29.874150 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:29.874154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:29.874155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:29.874166 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:29.876104 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:29.876721 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:29.876724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:29.876724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:29.876807 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:29.876994 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:29.876995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:29.876995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:29.877305 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:29.877474 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:29.877810 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:37:29.877939 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 19:37:29.878692 8671 catalog_manager.cc:1347] Generated new cluster ID: 48ed65b420654fe285b707f348ddf220
I20260924 19:37:29.878741 8671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:37:29.883244 8671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:37:29.883755 8671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:29.887583 8671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Generated new TSK 0
I20260924 19:37:29.887730 8671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:37:29.911859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:29.990214 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:29.990420 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:29.990451 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:29.991586 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:29.991626 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:29.991660 8684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:29.993145 8684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:29.993372 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:29.993577 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:29.998165 8690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:29.998229 8689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:29.998347 8684 server_base.cc:1034] running on GCE node
W20260924 19:37:29.998261 8692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:29.998580 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:29.998787 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:29.999966 8684 hybrid_clock.cc:648] HybridClock initialized: now 1790278649999916 us; error 27 us; skew 500 ppm
Sep 24 19:37:29 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.002370 8684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:37:30.002848 8684 webserver.cc:460] Webserver started at http://127.0.98.1:40385/ using document root <none> and password file <none>
I20260924 19:37:30.003059 8684 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.003108 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.003232 8684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:30.004376 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad979014c7f243be9f350f41a3effadb"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "3a9f2da0db7b9fcd9e4f95e177694d80"
server_key_iv: "e468c462b5a754635ffbb32ebeedf4e8"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.004702 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad979014c7f243be9f350f41a3effadb"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "3a9f2da0db7b9fcd9e4f95e177694d80"
server_key_iv: "e468c462b5a754635ffbb32ebeedf4e8"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.005909 8684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:37:30.006569 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.006709 8684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:30.006789 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "ad979014c7f243be9f350f41a3effadb"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "3a9f2da0db7b9fcd9e4f95e177694d80"
server_key_iv: "e468c462b5a754635ffbb32ebeedf4e8"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.006853 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.028915 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.030964 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.031100 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.031315 8684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.031627 8684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.031661 8684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.031688 8684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.031705 8684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.037130 8684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39721
I20260924 19:37:30.037184 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39721 every 8 connection(s)
I20260924 19:37:30.037505 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:30.045708 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.038145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57551 (local address 127.0.98.62:42509)
0924 19:37:30.038219 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.038222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.038396 (+ 174us) server_negotiation.cc:407] Connection header received
0924 19:37:30.039284 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.039285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.039294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.039342 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.039884 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.040380 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.041284 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.041473 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.041516 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.043378 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:30.043380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:30.043380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:30.043387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:30.044670 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.044996 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.044997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.044997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.045042 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.045205 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.045205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.045205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.045319 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:30.045363 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:30.045568 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:37:30.045641 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:37:30.045918 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.038129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:57551)
0924 19:37:30.038278 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.038318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.039205 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.039398 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.039401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.039635 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.039823 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.039828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.040438 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.040438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.041225 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.041228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.041526 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.041527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.041617 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.042002 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.042010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.043297 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:30.044728 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.044731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.044733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.044945 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.045083 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.045083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.045084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.045166 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.045361 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:30.045365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:30.045474 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:37:30.045549 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:37:30.045685 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260924 19:37:30.046351 8813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 19:37:30.046437 8813 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:30.046582 8813 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 19:37:30.046787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8684
I20260924 19:37:30.046886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:37:30.047077 392 external_mini_cluster.cc:1428] Setting key 10b5078af151b5e7b465bfcb5d4367aa
I20260924 19:37:30.047107 8622 ts_manager.cc:194] Registered new tserver with Master: ad979014c7f243be9f350f41a3effadb (127.0.98.1:39721)
I20260924 19:37:30.047915 8622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57551
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:37:30.068854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:30.149112 8818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.149313 8818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.149335 8818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.150498 8818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.150539 8818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.150568 8818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:30.152082 8818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.152289 8818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.152503 8818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.157143 8824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.157171 8823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.157202 8826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.157191 8818 server_base.cc:1034] running on GCE node
I20260924 19:37:30.157609 8818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.157850 8818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.159008 8818 hybrid_clock.cc:648] HybridClock initialized: now 1790278650158985 us; error 31 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.161487 8818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:37:30.161975 8818 webserver.cc:460] Webserver started at http://127.0.98.2:34159/ using document root <none> and password file <none>
I20260924 19:37:30.162192 8818 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.162243 8818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.162387 8818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:30.163463 8818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "e819cb1f00b24b8b93d93416dd4bf470"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "64332919d211acd40d20c32736fabe3d"
server_key_iv: "9fc71959b99bb99dd835ced683ee163e"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.163832 8818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e819cb1f00b24b8b93d93416dd4bf470"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "64332919d211acd40d20c32736fabe3d"
server_key_iv: "9fc71959b99bb99dd835ced683ee163e"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.165107 8818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:30.165787 8833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.165939 8818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:30.166018 8818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "e819cb1f00b24b8b93d93416dd4bf470"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "64332919d211acd40d20c32736fabe3d"
server_key_iv: "9fc71959b99bb99dd835ced683ee163e"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.166082 8818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.182443 8818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.184773 8818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.184926 8818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.185170 8818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.185494 8818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.185531 8818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.185564 8818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.185588 8818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.192368 8818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33919
I20260924 19:37:30.192410 8946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33919 every 8 connection(s)
I20260924 19:37:30.192746 8818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:37:30.193220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8818
I20260924 19:37:30.193318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:37:30.193507 392 external_mini_cluster.cc:1428] Setting key 4e190333f83b86fe270ae90d1cd09417
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:37:30.202769 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.193449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34307 (local address 127.0.98.62:42509)
0924 19:37:30.193517 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.193520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.193734 (+ 214us) server_negotiation.cc:407] Connection header received
0924 19:37:30.194776 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.194778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.194792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.194853 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.195603 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.196155 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.197568 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.197802 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.197863 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.200314 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:30.200315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:30.200316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:30.200322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:30.201555 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.201900 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.201901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.201901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.201952 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.202154 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.202155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.202155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.202287 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:30.202343 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:30.202597 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:37:30.202679 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:37:30.202934 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.193448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:34307)
0924 19:37:30.193612 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.193654 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.194675 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.194927 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.194932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.195217 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.195501 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.195508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.196241 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.196242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.197476 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.197479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.197894 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.197895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.198036 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.198560 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.198570 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.200216 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:30.201618 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.201625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.201627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.201837 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.202001 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.202002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.202002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.202101 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.202345 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:30.202351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:30.202484 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 19:37:30.202573 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:37:30.202723 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260924 19:37:30.203459 8947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 19:37:30.203552 8947 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 19:37:30.203743 8947 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 19:37:30.204144 8622 ts_manager.cc:194] Registered new tserver with Master: e819cb1f00b24b8b93d93416dd4bf470 (127.0.98.2:33919)
I20260924 19:37:30.204504 8622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:37:30.224813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:30.303117 8952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.303311 8952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.303333 8952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.304455 8952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.304492 8952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.304517 8952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:30.305943 8952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.306124 8952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.306367 8952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.311327 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.311398 8952 server_base.cc:1034] running on GCE node
W20260924 19:37:30.311323 8957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.311331 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.311676 8952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.311895 8952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.313064 8952 hybrid_clock.cc:648] HybridClock initialized: now 1790278650313052 us; error 25 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.315543 8952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:37:30.315958 8952 webserver.cc:460] Webserver started at http://127.0.98.3:45619/ using document root <none> and password file <none>
I20260924 19:37:30.316165 8952 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.316205 8952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.316318 8952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:30.317299 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "24669e6d82e34c02b1bb96b5a9703532"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "4e047f4c202b0f04d1aa36bbe261fb26"
server_key_iv: "2de0b2d2a58799b1bad9f987134ab0b2"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.317638 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "24669e6d82e34c02b1bb96b5a9703532"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "4e047f4c202b0f04d1aa36bbe261fb26"
server_key_iv: "2de0b2d2a58799b1bad9f987134ab0b2"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.318897 8952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:30.319541 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.319736 8952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:30.319819 8952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "24669e6d82e34c02b1bb96b5a9703532"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "4e047f4c202b0f04d1aa36bbe261fb26"
server_key_iv: "2de0b2d2a58799b1bad9f987134ab0b2"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.319885 8952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.331861 8952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.333917 8952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.334033 8952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.334235 8952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.334518 8952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.334550 8952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.334578 8952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.334599 8952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.339826 8952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36941
I20260924 19:37:30.339867 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36941 every 8 connection(s)
I20260924 19:37:30.340170 8952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:30.348347 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.340862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49913 (local address 127.0.98.62:42509)
0924 19:37:30.340903 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.340905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.341085 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:37:30.341982 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.341982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.341991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.342046 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.342604 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.343091 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.344052 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.344259 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.344310 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.346135 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:30.346136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:30.346137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:30.346143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:30.347289 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.347609 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.347610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.347610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.347652 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.347829 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.347829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.347830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.347918 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:30.347996 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:30.348226 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 19:37:30.348281 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:37:30.348523 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.340826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:49913)
0924 19:37:30.340973 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.341013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.341902 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.342109 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.342113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.342359 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.342544 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.342548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.343150 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.343151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.343986 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.343988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.344329 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.344330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.344439 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.344839 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.344846 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.346054 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:30.347358 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.347364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.347366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.347556 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.347697 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:30.347697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:30.347698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:30.347780 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:30.347998 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:30.348004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:30.348114 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:37:30.348212 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:37:30.348325 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 19:37:30.348946 9081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 19:37:30.349027 9081 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:30.349182 9081 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 19:37:30.349239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 8952
I20260924 19:37:30.349340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:37:30.349531 392 external_mini_cluster.cc:1428] Setting key 642e55660a01252efb801c91c84bd10c
I20260924 19:37:30.349543 8622 ts_manager.cc:194] Registered new tserver with Master: 24669e6d82e34c02b1bb96b5a9703532 (127.0.98.3:36941)
I20260924 19:37:30.349959 8622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49913
I20260924 19:37:30.350488 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:37:30.411576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8684
I20260924 19:37:30.415632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8818
I20260924 19:37:30.419649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8952
I20260924 19:37:30.423328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 8590
I20260924 19:37:30.427176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--webserver_interface=127.0.98.62
--webserver_port=32787
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:30.505429 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.505632 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.505662 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.506827 9090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:30.506867 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.506886 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.506902 9090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:30.506917 9090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:30.508419 9090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.508644 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.508848 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.513252 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.513242 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.513242 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.513468 9090 server_base.cc:1034] running on GCE node
I20260924 19:37:30.513685 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.513928 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.515061 9090 hybrid_clock.cc:648] HybridClock initialized: now 1790278650515048 us; error 26 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.517376 9090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:37:30.517742 9090 webserver.cc:460] Webserver started at http://127.0.98.62:32787/ using document root <none> and password file <none>
I20260924 19:37:30.517925 9090 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.517971 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.519186 9090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:30.519758 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.519883 9090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:30.519979 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "6d34aec2c68c47fba54f61c04da7f2af"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "98302deff0b81aecfb597785ed093585"
server_key_iv: "068046bc5ae4a670d9b66f5881bf25df"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.520282 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.528160 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.530026 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.530123 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.533689 9090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42509
I20260924 19:37:30.533720 9157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42509 every 8 connection(s)
I20260924 19:37:30.534183 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:30.535353 9158 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:37:30.535985 9158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Bootstrap starting.
I20260924 19:37:30.540962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9090
I20260924 19:37:30.541234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39721
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42509
--webserver_port=40385
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:37:30.543996 9158 log.cc:824] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:30.545568 9158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:37:30.545805 9158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Bootstrap complete.
I20260924 19:37:30.547833 9158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:30.548096 9158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34aec2c68c47fba54f61c04da7f2af, State: Initialized, Role: FOLLOWER
I20260924 19:37:30.548259 9158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:30.548331 9158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:30.548375 9158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:30.548419 9158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:37:30.549332 9158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:30.549425 9158 leader_election.cc:302] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d34aec2c68c47fba54f61c04da7f2af; no voters:
I20260924 19:37:30.549568 9158 leader_election.cc:288] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:37:30.549620 9162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:37:30.549733 9162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [term 2 LEADER]: Becoming Leader. State: Replica: 6d34aec2c68c47fba54f61c04da7f2af, State: Running, Role: LEADER
I20260924 19:37:30.549816 9158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:30.549814 9162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 19:37:30.550102 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6d34aec2c68c47fba54f61c04da7f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 19:37:30.550222 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:30.550109 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d34aec2c68c47fba54f61c04da7f2af. Latest consensus state: current_term: 2 leader_uuid: "6d34aec2c68c47fba54f61c04da7f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d34aec2c68c47fba54f61c04da7f2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 19:37:30.550312 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:30.550565 9171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:30.551014 9171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:30.551126 9171 catalog_manager.cc:1259] Loaded cluster ID: 48ed65b420654fe285b707f348ddf220
I20260924 19:37:30.551154 9171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:37:30.552502 9171 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:30.552676 9171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6d34aec2c68c47fba54f61c04da7f2af: Loaded TSK: 0
I20260924 19:37:30.552879 9171 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:37:30.620026 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.620206 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.620227 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.621364 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.621404 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.621428 9160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:30.622845 9160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40385
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.623047 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.623245 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.627791 9184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.627841 9187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.627967 9185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.628111 9160 server_base.cc:1034] running on GCE node
I20260924 19:37:30.628306 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.628559 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.629731 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790278650629709 us; error 34 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.632618 9160 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:37:30.633119 9160 webserver.cc:460] Webserver started at http://127.0.98.1:40385/ using document root <none> and password file <none>
I20260924 19:37:30.633381 9160 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.633455 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.635035 9160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:30.635618 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.635767 9160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:30.635859 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "ad979014c7f243be9f350f41a3effadb"
format_stamp: "Formatted at 2026-09-24 19:37:29 on dist-test-slave-6kf3"
server_key: "3a9f2da0db7b9fcd9e4f95e177694d80"
server_key_iv: "e468c462b5a754635ffbb32ebeedf4e8"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.636277 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.659677 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.662189 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.662331 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.662551 9160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.662863 9160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.662899 9160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.662930 9160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.662948 9160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.668488 9160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39721
I20260924 19:37:30.668540 9307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39721 every 8 connection(s)
I20260924 19:37:30.668838 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.676342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9160
I20260924 19:37:30.676628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42509
--webserver_port=34159
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:37:30.676710 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.669525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:33007)
0924 19:37:30.669670 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.669712 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.670192 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.670585 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.670589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.671133 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.671324 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.671329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.672445 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.672446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.673219 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.673221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.673761 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.673762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.673889 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.674362 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.674370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.676515 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260924 19:37:30.677273 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.669602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33007 (local address 127.0.98.62:42509)
0924 19:37:30.669786 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.669794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.669841 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:37:30.670388 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.670392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.670433 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.670544 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.671434 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.672351 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.673295 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.673686 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.673781 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.676654 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33007: BlockingRecv error: recv got EOF from 127.0.98.1:33007 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
W20260924 19:37:30.677660 9308 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42509: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.682835 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.677871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:45197)
0924 19:37:30.677969 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.677978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.678027 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.678174 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.678175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.678329 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.678387 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.678389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.679211 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.679212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.679891 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.679894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.680398 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.680399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.680461 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.680915 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.680921 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.682785 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 19:37:30.682947 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.677925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45197 (local address 127.0.98.62:42509)
0924 19:37:30.677963 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.677965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.678014 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:37:30.678077 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.678077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.678090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.678150 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.678455 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.679127 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.680021 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.680319 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.680384 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.682887 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45197: BlockingRecv error: recv got EOF from 127.0.98.1:45197 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.687433 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.683064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:55543)
0924 19:37:30.683209 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.683217 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.683255 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.683373 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.683374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.683472 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.683499 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.683500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.684365 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.684373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.684873 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.684875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.685296 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.685296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.685332 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.685785 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.685789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.687381 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260924 19:37:30.687528 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.683119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55543 (local address 127.0.98.62:42509)
0924 19:37:30.683152 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.683154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.683259 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:37:30.683287 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.683288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.683298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.683360 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.683550 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.684272 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.684954 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.685224 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.685287 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.687469 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55543: BlockingRecv error: recv got EOF from 127.0.98.1:55543 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 19:37:30.687562 9308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:37:30.756487 9312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.756675 9312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.756695 9312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.757783 9312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.757822 9312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.757845 9312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:30.759164 9312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34159
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.759333 9312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.759498 9312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.763916 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.764006 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.763895 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.764063 9312 server_base.cc:1034] running on GCE node
I20260924 19:37:30.764290 9312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.764490 9312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.765614 9312 hybrid_clock.cc:648] HybridClock initialized: now 1790278650765584 us; error 47 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.767800 9312 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:37:30.768224 9312 webserver.cc:460] Webserver started at http://127.0.98.2:34159/ using document root <none> and password file <none>
I20260924 19:37:30.768433 9312 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.768493 9312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.769706 9312 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:30.770207 9328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.770346 9312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.770418 9312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "e819cb1f00b24b8b93d93416dd4bf470"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "64332919d211acd40d20c32736fabe3d"
server_key_iv: "9fc71959b99bb99dd835ced683ee163e"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.770710 9312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.779397 9312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.781376 9312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.781497 9312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.781678 9312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.781970 9312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.782009 9312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.782037 9312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.782059 9312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.787256 9312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33919
I20260924 19:37:30.787305 9441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33919 every 8 connection(s)
I20260924 19:37:30.787606 9312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:37:30.791440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9312
I20260924 19:37:30.791700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36941
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42509
--webserver_port=45619
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:37:30.794248 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.788281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57019 (local address 127.0.98.62:42509)
0924 19:37:30.788349 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.788351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.788529 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:37:30.789007 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.789008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.789021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.789064 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.789893 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.790368 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.791288 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.791464 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.791513 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.794193 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57019: BlockingRecv error: recv got EOF from 127.0.98.2:57019 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:37:30.794276 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.788254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:57019)
0924 19:37:30.788419 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.788454 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.788935 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.789126 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.789129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.789644 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.789834 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.789839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.790433 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.790434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.791233 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.791235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.791546 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.791548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.791655 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.792148 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.792155 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.794098 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
W20260924 19:37:30.794456 9442 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42509: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.798381 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.794639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:33777)
0924 19:37:30.794675 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.794684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.794714 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.794847 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.794847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.794958 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.794994 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.794995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.795556 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.795557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.796109 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.796110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.796417 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.796418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.796473 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.796796 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.796801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.798328 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:37:30.798449 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.794702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33777 (local address 127.0.98.62:42509)
0924 19:37:30.794756 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.794758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.794765 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:37:30.794785 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.794786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.794795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.794839 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.795053 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.795496 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.796179 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.796367 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.796414 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.798404 (+ 1990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33777: BlockingRecv error: recv got EOF from 127.0.98.2:33777 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.802685 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.798714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:53895)
0924 19:37:30.798807 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.798817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.798848 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.798958 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.798959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.799051 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.799073 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.799074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.799864 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.799864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.800358 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.800359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.800790 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.800791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.800824 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.801130 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.801134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.802631 (+ 1497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 19:37:30.802915 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.798770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53895 (local address 127.0.98.62:42509)
0924 19:37:30.798827 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.798829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.798840 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:37:30.798887 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.798888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.798897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.798957 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.799124 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.799799 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.800422 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.800730 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.800794 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.802855 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53895: BlockingRecv error: recv got EOF from 127.0.98.2:53895 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 19:37:30.803741 9442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:37:30.871330 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:30.871523 9445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:30.871544 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:30.872692 9445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:30.872731 9445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:30.872756 9445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:30.874188 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45619
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:30.874361 9445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:30.874527 9445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:30.879129 9450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.879258 9451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:30.879127 9453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:30.879297 9445 server_base.cc:1034] running on GCE node
I20260924 19:37:30.879556 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:30.879777 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:30.880918 9445 hybrid_clock.cc:648] HybridClock initialized: now 1790278650880888 us; error 40 us; skew 500 ppm
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278650, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:30.883132 9445 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:37:30.883549 9445 webserver.cc:460] Webserver started at http://127.0.98.3:45619/ using document root <none> and password file <none>
I20260924 19:37:30.883749 9445 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:30.883800 9445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:30.885100 9445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:30.885768 9460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.885905 9445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:30.885977 9445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "24669e6d82e34c02b1bb96b5a9703532"
format_stamp: "Formatted at 2026-09-24 19:37:30 on dist-test-slave-6kf3"
server_key: "4e047f4c202b0f04d1aa36bbe261fb26"
server_key_iv: "2de0b2d2a58799b1bad9f987134ab0b2"
server_key_version: "encryptionkey@0"
I20260924 19:37:30.886296 9445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:30.894263 9445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:30.896000 9445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:30.896116 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:30.896304 9445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:30.896564 9445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:30.896593 9445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.896621 9445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:30.896644 9445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:30.901959 9445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36941
I20260924 19:37:30.902011 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36941 every 8 connection(s)
I20260924 19:37:30.902298 9445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:37:30.905936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9445
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:37:30.909523 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.902925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54239 (local address 127.0.98.62:42509)
0924 19:37:30.902972 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.902974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.903149 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:37:30.903676 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.903677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.903690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.903740 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.904660 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.905147 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.906419 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.906656 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.906713 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.909466 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54239: BlockingRecv error: recv got EOF from 127.0.98.3:54239 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:37:30.909586 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.902907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:54239)
0924 19:37:30.903032 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.903070 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.903576 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.903806 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.903810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.904379 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.904564 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.904568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.905224 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.905225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.906208 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.906210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.906730 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.906731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.906859 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.907309 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.907316 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.909400 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
W20260924 19:37:30.909754 9574 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42509: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 19:37:30.912853 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.906429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.0.98.62:42509)
0924 19:37:30.906549 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.906551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.906570 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:37:30.906591 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.906592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.906603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.906652 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.907232 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.907695 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.908611 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.908821 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.908868 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.910096 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:30.910100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:30.910100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:30.910129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:30.911590 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.911893 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.911896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.911897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.912036 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:30.912207 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:30.912208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:30.912208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:30.912431 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:30.912559 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:30.912649 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:37:30.912757 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.914040 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.909931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:60209)
0924 19:37:30.910019 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.910028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.910071 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.910169 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.910170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.910281 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.910328 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.910329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.911113 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.911114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.911620 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.911622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.911963 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.911964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.912024 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.912346 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.912351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.913982 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 19:37:30.914104 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.909982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60209 (local address 127.0.98.62:42509)
0924 19:37:30.910023 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.910025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.910057 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:37:30.910121 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.910122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.910130 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.910168 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.910398 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.911035 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.911696 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.911889 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.911965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.914059 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60209: BlockingRecv error: recv got EOF from 127.0.98.3:60209 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:30 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:30.917718 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:30.914308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:33971)
0924 19:37:30.914345 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:30.914354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:30.914387 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:30.914476 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:30.914476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:30.914566 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:30.914589 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.914590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.915094 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.915095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.915572 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:30.915573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.915926 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.915926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:30.916003 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.916350 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:30.916354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:30.917676 (+ 1322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:37:30.917821 9574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:37:30.917837 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:30.914356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33971 (local address 127.0.98.62:42509)
0924 19:37:30.914391 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:30.914393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:30.914416 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:37:30.914430 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:30.914431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:30.914437 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:30.914467 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:30.914624 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.915043 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.915634 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:30.915857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:30.915907 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:30.917801 (+ 1894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33971: BlockingRecv error: recv got EOF from 127.0.98.3:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:31.692340 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:31.687963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:34415)
0924 19:37:31.688071 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:31.688081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:31.688133 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:31.688296 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:31.688297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:31.688449 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:31.688495 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.688496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.689173 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.689174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.689764 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.689766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.690108 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.690109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.690143 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.690566 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:31.690571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:31.692274 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260924 19:37:31.692418 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:31.688021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34415 (local address 127.0.98.62:42509)
0924 19:37:31.688118 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:31.688121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:31.688128 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:37:31.688190 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:31.688190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:31.688199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:31.688255 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:31.688563 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.689091 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.689846 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.690055 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.690104 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.692359 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34415: BlockingRecv error: recv got EOF from 127.0.98.1:34415 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:31.808269 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:31.804086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:57621)
0924 19:37:31.804197 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:31.804209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:31.804272 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:31.804389 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:31.804389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:31.804551 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:31.804617 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.804619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.805255 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.805256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.805790 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.805791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.806125 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.806126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.806189 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.806609 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:31.806614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:31.808184 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260924 19:37:31.808326 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:31.804156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57621 (local address 127.0.98.62:42509)
0924 19:37:31.804216 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:31.804218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:31.804254 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:37:31.804321 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:31.804322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:31.804333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:31.804375 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:31.804687 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.805185 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.805868 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.806068 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.806121 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.808277 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57621: BlockingRecv error: recv got EOF from 127.0.98.2:57621 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:31 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:31.922883 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:31.918258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:49663)
0924 19:37:31.918412 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:31.918425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:31.918494 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:31.918638 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:31.918639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:31.918791 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:31.918836 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.918837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.919488 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.919490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.920093 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:31.920095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.920471 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.920471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:31.920512 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.921000 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:31.921005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:31.922817 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
W20260924 19:37:31.922966 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:31.918359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49663 (local address 127.0.98.62:42509)
0924 19:37:31.918412 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:31.918414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:31.918492 (+ 78us) server_negotiation.cc:407] Connection header received
0924 19:37:31.918557 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:31.918557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:31.918571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:31.918622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:31.918900 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.919396 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.920199 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:31.920402 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:31.920462 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:31.922908 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49663: BlockingRecv error: recv got EOF from 127.0.98.3:49663 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:32.700443 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:32.692796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:53773)
0924 19:37:32.692940 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:32.692949 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:32.692999 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:32.693147 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:32.693148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:32.693301 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:32.693358 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.693359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.693977 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.693978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.694520 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.694522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.694832 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.694832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.694867 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.698707 (+ 3840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:32.698712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:32.700378 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260924 19:37:32.700567 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:32.692888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53773 (local address 127.0.98.62:42509)
0924 19:37:32.693022 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:32.693024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:32.693034 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:37:32.693055 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:32.693055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:32.693065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:32.693128 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:32.693417 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.693921 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.694583 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.694780 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.694831 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.700491 (+ 5660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53773: BlockingRecv error: recv got EOF from 127.0.98.1:53773 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:32.813246 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:32.808676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:35585)
0924 19:37:32.808820 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:32.808832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:32.808899 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:32.809192 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:32.809193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:32.809373 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:32.809423 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.809425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.810110 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.810111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.810631 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.810632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.810968 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.810969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.811023 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.811438 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:32.811442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:32.813170 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
W20260924 19:37:32.813297 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:32.808807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35585 (local address 127.0.98.62:42509)
0924 19:37:32.808867 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:32.808869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:32.808880 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:37:32.809104 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:32.809105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:32.809120 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:32.809196 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:32.809497 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.810019 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.810719 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.810908 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.810966 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.813251 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35585: BlockingRecv error: recv got EOF from 127.0.98.2:35585 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:32 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:32.927527 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:32.923368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:41183)
0924 19:37:32.923479 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:32.923489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:32.923548 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:32.923668 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:32.923668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:32.923834 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:32.923903 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.923905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.924615 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.924616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.925139 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:32.925141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.925458 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.925458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:32.925510 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.925940 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:32.925944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:32.927453 (+ 1509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
W20260924 19:37:32.927594 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:32.923498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41183 (local address 127.0.98.62:42509)
0924 19:37:32.923559 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:32.923560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:32.923571 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:37:32.923597 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:32.923598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:32.923611 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:32.923657 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:32.924021 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.924547 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.925211 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:32.925401 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:32.925454 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:32.927542 (+ 2088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41183: BlockingRecv error: recv got EOF from 127.0.98.3:41183 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:33.705401 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:33.700890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:37347)
0924 19:37:33.701017 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:33.701028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:33.701081 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:33.701218 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:33.701219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:33.701375 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:33.701423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.701424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.702042 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.702043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.702566 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.702567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.702885 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.702886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.702926 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.703329 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:33.703334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:33.705287 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
W20260924 19:37:33.705430 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:33.700959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37347 (local address 127.0.98.62:42509)
0924 19:37:33.701097 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:33.701099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:33.701108 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:33.701131 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:33.701132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:33.701141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:33.701205 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:33.701485 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.701983 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.702638 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.702825 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.702874 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.705369 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37347: BlockingRecv error: recv got EOF from 127.0.98.1:37347 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:33.818037 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:33.813686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:37297)
0924 19:37:33.813831 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:33.813841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:33.813897 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:33.814020 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:33.814020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:33.814171 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:33.814220 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.814222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.814903 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.814905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.815437 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.815439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.815787 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.815788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.815834 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.816280 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:33.816284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:33.817980 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
W20260924 19:37:33.818150 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:33.813754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37297 (local address 127.0.98.62:42509)
0924 19:37:33.813808 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:33.813810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:33.813894 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:37:33.813953 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:33.813954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:33.813965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:33.814001 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:33.814309 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.814826 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.815517 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.815714 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.815782 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.818089 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37297: BlockingRecv error: recv got EOF from 127.0.98.2:37297 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:33 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:33.932372 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:33.928011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:39937)
0924 19:37:33.928162 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:33.928172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:33.928235 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:33.928358 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:33.928359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:33.928531 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:33.928576 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.928578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.929204 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.929205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.929729 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:33.929731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.930046 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.930047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:33.930099 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.930535 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:33.930541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:33.932288 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
W20260924 19:37:33.932441 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:33.928067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39937 (local address 127.0.98.62:42509)
0924 19:37:33.928119 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:33.928122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:33.928228 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:37:33.928287 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:33.928288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:33.928303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:33.928348 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:33.928644 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.929133 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.929805 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:33.929989 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:33.930043 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:33.932387 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39937: BlockingRecv error: recv got EOF from 127.0.98.3:39937 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:34.710481 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:34.705926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:34365)
0924 19:37:34.706098 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:34.706111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:34.706170 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:34.706295 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:34.706296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:34.706455 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:34.706501 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.706502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.707125 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.707126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.707640 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.707641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.708064 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.708065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.708117 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.708597 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:34.708602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:34.710398 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
W20260924 19:37:34.710572 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:34.705997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34365 (local address 127.0.98.62:42509)
0924 19:37:34.706159 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:34.706160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:34.706171 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:37:34.706223 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:34.706224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:34.706234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:34.706281 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:34.706563 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.707063 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.707741 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.707967 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.708035 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.710506 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34365: BlockingRecv error: recv got EOF from 127.0.98.1:34365 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:34.823223 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:34.818579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:50899)
0924 19:37:34.818722 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:34.818734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:34.818818 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:34.818985 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:34.818986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:34.819144 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:34.819189 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.819191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.819974 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.819975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.820520 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.820522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.820856 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.820857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.820898 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.821384 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:34.821389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:34.823125 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260924 19:37:34.823267 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:34.818698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50899 (local address 127.0.98.62:42509)
0924 19:37:34.818745 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:34.818746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:34.818797 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:37:34.818883 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:34.818885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:34.818895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:34.818934 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:34.819294 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.819868 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.820605 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.820796 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.820850 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.823216 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50899: BlockingRecv error: recv got EOF from 127.0.98.2:50899 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:34 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:34.937036 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:34.932882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:51493)
0924 19:37:34.933022 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:34.933032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:34.933088 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:34.933210 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:34.933211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:34.933367 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:34.933413 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.933415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.934050 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.934050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.934556 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:34.934557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.934864 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.934864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:34.934899 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.935315 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:34.935319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:34.936976 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
W20260924 19:37:34.937136 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:34.932966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51493 (local address 127.0.98.62:42509)
0924 19:37:34.933020 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:34.933022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:34.933076 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:37:34.933144 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:34.933144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:34.933156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:34.933195 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:34.933483 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.933992 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.934628 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:34.934809 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:34.934860 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:34.937074 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51493: BlockingRecv error: recv got EOF from 127.0.98.3:51493 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:35.715415 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:35.710919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:49705)
0924 19:37:35.711062 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:35.711073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:35.711126 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:35.711291 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:35.711292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:35.711448 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:35.711495 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.711497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.712178 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.712179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.712777 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.712780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.713150 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.713151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.713201 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.713659 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:35.713664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:35.715341 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
W20260924 19:37:35.715494 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:35.711003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49705 (local address 127.0.98.62:42509)
0924 19:37:35.711110 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:35.711112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:35.711121 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:35.711174 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:35.711175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:35.711184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:35.711242 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:35.711557 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.712090 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.712868 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.713066 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.713120 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.715433 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49705: BlockingRecv error: recv got EOF from 127.0.98.1:49705 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:35.827770 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:35.823672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:55215)
0924 19:37:35.823844 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:35.823855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:35.823925 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:35.824065 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:35.824066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:35.824216 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:35.824267 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.824269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.824872 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.824874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.825403 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.825404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.825694 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.825696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.825736 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.826146 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:35.826150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:35.827706 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
W20260924 19:37:35.827860 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:35.823776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55215 (local address 127.0.98.62:42509)
0924 19:37:35.823837 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:35.823839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:35.823912 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:37:35.823994 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:35.823995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:35.824004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:35.824054 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:35.824316 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.824801 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.825461 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.825638 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.825684 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.827800 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55215: BlockingRecv error: recv got EOF from 127.0.98.2:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:35 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:35.942344 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:35.937534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:59027)
0924 19:37:35.937691 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:35.937705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:35.937783 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:35.937939 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:35.937940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:35.938114 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:35.938163 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.938164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.938832 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.938833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.939451 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:35.939455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.939986 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.939987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:35.940049 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.940539 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:35.940544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:35.942287 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
W20260924 19:37:35.942446 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:35.937652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59027 (local address 127.0.98.62:42509)
0924 19:37:35.937709 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:35.937711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:35.937763 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:37:35.937855 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:35.937856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:35.937873 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:35.937928 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:35.938238 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.938753 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.939529 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:35.939877 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:35.939966 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:35.942384 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59027: BlockingRecv error: recv got EOF from 127.0.98.3:59027 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:36.720736 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:36.715919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:47265)
0924 19:37:36.716074 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:36.716088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:36.716157 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:36.716470 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:36.716471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:36.716623 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:36.716673 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.716675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.717369 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.717370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.717959 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.717961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.718318 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.718318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.718354 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.718843 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:36.718846 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:36.720652 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260924 19:37:36.720834 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:36.716052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47265 (local address 127.0.98.62:42509)
0924 19:37:36.716233 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:36.716235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:36.716247 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:37:36.716276 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:36.716276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:36.716291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:36.716449 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:36.716789 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.717301 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.718045 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.718252 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.718323 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.720768 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47265: BlockingRecv error: recv got EOF from 127.0.98.1:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:36.832935 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:36.828277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:60769)
0924 19:37:36.828403 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:36.828413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:36.828464 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:36.828597 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:36.828598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:36.828754 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:36.828838 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.828840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.829515 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.829516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.830244 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.830247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.830665 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.830666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.830708 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.831197 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:36.831201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:36.832869 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
W20260924 19:37:36.833021 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:36.828366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60769 (local address 127.0.98.62:42509)
0924 19:37:36.828442 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:36.828443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:36.828455 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:37:36.828518 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:36.828519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:36.828530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:36.828586 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:36.828903 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.829439 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.830352 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.830598 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.830658 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.832963 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60769: BlockingRecv error: recv got EOF from 127.0.98.2:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:36 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:36.947310 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:36.942849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:43291)
0924 19:37:36.942982 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:36.942993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:36.943055 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:36.943182 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:36.943183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:36.943331 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:36.943377 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.943378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.944044 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.944045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.944592 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:36.944593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.944924 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.944925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:36.944960 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.945400 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:36.945405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:36.947221 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20260924 19:37:36.947378 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:36.942912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43291 (local address 127.0.98.62:42509)
0924 19:37:36.942988 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:36.942990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:36.943043 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:37:36.943113 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:36.943114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:36.943128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:36.943171 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:36.943453 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.943987 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.944661 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:36.944864 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:36.944924 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:36.947324 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43291: BlockingRecv error: recv got EOF from 127.0.98.3:43291 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:37.729240 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:37.721225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:41207)
0924 19:37:37.721341 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:37.721353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:37.721403 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:37.721575 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:37.721575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:37.721729 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:37.721778 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.721780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.722524 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.722526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.723072 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.723075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.723427 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.723428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.723474 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.727251 (+ 3777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:37.727257 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:37.729181 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260924 19:37:37.729341 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:37.721316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41207 (local address 127.0.98.62:42509)
0924 19:37:37.721453 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:37.721455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:37.721463 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:37.721484 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:37.721485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:37.721494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:37.721557 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:37.721892 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.722425 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.723163 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.723362 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.723417 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.729273 (+ 5856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41207: BlockingRecv error: recv got EOF from 127.0.98.1:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:37.837603 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:37.833389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:40759)
0924 19:37:37.833537 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:37.833549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:37.833611 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:37.833738 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:37.833739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:37.833930 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:37.833977 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.833979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.834579 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.834580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.835081 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.835082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.835433 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.835433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.835469 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.835857 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:37.835861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:37.837526 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
W20260924 19:37:37.837672 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:37.833457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40759 (local address 127.0.98.62:42509)
0924 19:37:37.833504 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:37.833507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:37.833604 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:37:37.833663 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:37.833664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:37.833676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:37.833725 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:37.834032 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.834524 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.835171 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.835374 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.835432 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.837621 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40759: BlockingRecv error: recv got EOF from 127.0.98.2:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:37 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:37.952404 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:37.947822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:44677)
0924 19:37:37.947991 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:37.948002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:37.948055 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:37.948184 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:37.948185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:37.948333 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:37.948378 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.948379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.949000 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.949000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.949559 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:37.949562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.949985 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.949986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:37.950028 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.950497 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:37.950501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:37.952346 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
W20260924 19:37:37.952485 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:37.947956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44677 (local address 127.0.98.62:42509)
0924 19:37:37.948008 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:37.948010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:37.948042 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:37:37.948115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:37.948116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:37.948129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:37.948174 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:37.948437 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.948942 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.949635 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:37.949880 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:37.949971 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:37.952425 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44677: BlockingRecv error: recv got EOF from 127.0.98.3:44677 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:38.733789 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:38.729663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:48971)
0924 19:37:38.729808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:38.729820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:38.729872 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:38.730025 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:38.730030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:38.730185 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:38.730233 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.730234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.730853 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.730855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.731395 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.731397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.731716 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.731717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.731753 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.732173 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:38.732177 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:38.733732 (+ 1555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
W20260924 19:37:38.733881 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:38.729778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48971 (local address 127.0.98.62:42509)
0924 19:37:38.729878 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:38.729879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:38.729889 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:37:38.729910 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:38.729910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:38.729920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:38.729981 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:38.730292 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.730779 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.731471 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.731662 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.731714 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.733814 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48971: BlockingRecv error: recv got EOF from 127.0.98.1:48971 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:38.842523 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:38.838059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:59291)
0924 19:37:38.838196 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:38.838206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:38.838265 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:38.838390 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:38.838391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:38.838546 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:38.838587 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.838589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.839386 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.839387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.839992 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.839995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.840323 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.840324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.840365 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.840795 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:38.840799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:38.842462 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
W20260924 19:37:38.842645 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:38.838134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59291 (local address 127.0.98.62:42509)
0924 19:37:38.838182 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:38.838184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:38.838250 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:37:38.838325 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:38.838326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:38.838336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:38.838380 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:38.838671 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.839326 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.840056 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.840265 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.840324 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.842585 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59291: BlockingRecv error: recv got EOF from 127.0.98.2:59291 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:38 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:38.957234 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:38.952834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:60955)
0924 19:37:38.952971 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:38.952984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:38.953062 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:38.953198 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:38.953199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:38.953373 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:38.953424 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.953426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.954162 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.954164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.954806 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:38.954809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.955120 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.955120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:38.955160 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.955564 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:38.955568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:38.957177 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
W20260924 19:37:38.957317 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:38.952925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60955 (local address 127.0.98.62:42509)
0924 19:37:38.953008 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:38.953010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:38.953040 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:37:38.953119 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:38.953120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:38.953134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:38.953187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:38.953492 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.954074 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.954877 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:38.955062 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:38.955116 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:38.957259 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60955: BlockingRecv error: recv got EOF from 127.0.98.3:60955 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:39.738684 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:39.734271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:56489)
0924 19:37:39.734415 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:39.734427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:39.734483 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:39.734638 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:39.734639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:39.734807 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:39.734856 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.734858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.735499 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.735500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.736049 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.736051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.736364 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.736365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.736399 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.736822 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:39.736826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:39.738608 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
W20260924 19:37:39.738765 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:39.734368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56489 (local address 127.0.98.62:42509)
0924 19:37:39.734489 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:39.734490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:39.734502 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:37:39.734525 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:39.734526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:39.734536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:39.734594 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:39.734920 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.735431 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.736114 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.736319 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.736362 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.738702 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56489: BlockingRecv error: recv got EOF from 127.0.98.1:56489 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:39.847616 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:39.842979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:53353)
0924 19:37:39.843112 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:39.843122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:39.843186 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:39.843353 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:39.843355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:39.843512 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:39.843557 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.843559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.844458 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.844460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.845027 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.845029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.845368 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.845369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.845422 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.845908 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:39.845913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:39.847540 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260924 19:37:39.847684 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:39.843048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53353 (local address 127.0.98.62:42509)
0924 19:37:39.843116 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:39.843118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:39.843180 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:37:39.843250 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:39.843251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:39.843263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:39.843303 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:39.843651 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.844386 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.845099 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.845306 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.845361 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.847635 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53353: BlockingRecv error: recv got EOF from 127.0.98.2:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:39 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:39.961930 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:39.957665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:37501)
0924 19:37:39.957778 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:39.957790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:39.957859 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:39.958012 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:39.958012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:39.958168 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:39.958212 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.958214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.958841 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.958842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.959352 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:39.959354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.959674 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.959675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:39.959729 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.960198 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:39.960202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:39.961874 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
W20260924 19:37:39.962013 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:39.957735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37501 (local address 127.0.98.62:42509)
0924 19:37:39.957795 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:39.957797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:39.957850 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:37:39.957914 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:39.957915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:39.957929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:39.957994 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:39.958280 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.958773 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.959423 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:39.959616 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:39.959668 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:39.961956 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37501: BlockingRecv error: recv got EOF from 127.0.98.3:37501 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:40.743331 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:40.739147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:43151)
0924 19:37:40.739260 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:40.739270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:40.739325 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:40.739437 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:40.739438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:40.739593 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:40.739639 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.739641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.740357 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.740358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.740901 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.740903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.741270 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.741270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.741325 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.741711 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:40.741715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:40.743257 (+ 1542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
W20260924 19:37:40.743405 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:40.739229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43151 (local address 127.0.98.62:42509)
0924 19:37:40.739330 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:40.739332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:40.739341 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:40.739362 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:40.739363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:40.739373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:40.739427 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:40.739711 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.740265 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.740990 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.741211 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.741265 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.743344 (+ 2079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43151: BlockingRecv error: recv got EOF from 127.0.98.1:43151 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:40.852751 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:40.848147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:35141)
0924 19:37:40.848283 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:40.848310 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:40.848373 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:40.848540 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:40.848541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:40.848697 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:40.848742 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.848743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.849424 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.849425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.850035 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.850037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.850401 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.850401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.850442 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.850848 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:40.850852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:40.852693 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
W20260924 19:37:40.852839 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:40.848236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35141 (local address 127.0.98.62:42509)
0924 19:37:40.848287 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:40.848289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:40.848365 (+ 76us) server_negotiation.cc:407] Connection header received
0924 19:37:40.848416 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:40.848416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:40.848426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:40.848498 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:40.848809 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.849321 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.850139 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.850342 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.850392 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.852779 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35141: BlockingRecv error: recv got EOF from 127.0.98.2:35141 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:40 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:40.966840 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:40.962450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:59287)
0924 19:37:40.962586 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:40.962596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:40.962654 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:40.962784 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:40.962784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:40.962954 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:40.962999 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.963001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.963638 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.963639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.964189 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:40.964192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.964495 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.964496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:40.964534 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.964976 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:40.964981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:40.966778 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
W20260924 19:37:40.966949 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:40.962562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59287 (local address 127.0.98.62:42509)
0924 19:37:40.962617 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:40.962619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:40.962643 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:37:40.962704 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:40.962705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:40.962716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:40.962752 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:40.963051 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.963532 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.964250 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:40.964443 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:40.964488 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:40.966883 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59287: BlockingRecv error: recv got EOF from 127.0.98.3:59287 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:41.748328 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:41.743813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:41243)
0924 19:37:41.743999 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:41.744012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:41.744078 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:41.744203 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:41.744203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:41.744361 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:41.744410 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.744411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.745051 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.745053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.745596 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.745598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.745990 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.745992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.746048 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.746527 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:41.746532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:41.748250 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
W20260924 19:37:41.748402 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:41.743892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41243 (local address 127.0.98.62:42509)
0924 19:37:41.744039 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:41.744041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:41.744061 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:37:41.744136 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:41.744136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:41.744146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:41.744191 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:41.744460 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.744963 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.745690 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.745890 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.745954 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.748343 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41243: BlockingRecv error: recv got EOF from 127.0.98.1:41243 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:41.857378 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:41.853152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:33597)
0924 19:37:41.853269 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:41.853279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:41.853345 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:41.853460 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:41.853461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:41.853616 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:41.853661 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.853662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.854293 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.854294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.854852 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.854855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.855161 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.855162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.855206 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.855640 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:41.855644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:41.857308 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
W20260924 19:37:41.857465 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:41.853205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33597 (local address 127.0.98.62:42509)
0924 19:37:41.853273 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:41.853275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:41.853341 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:37:41.853399 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:41.853400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:41.853410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:41.853446 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:41.853710 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.854207 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.854918 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.855107 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.855149 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.857405 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33597: BlockingRecv error: recv got EOF from 127.0.98.2:33597 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:41 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:41.971756 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:41.967268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:60929)
0924 19:37:41.967387 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:41.967399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:41.967463 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:41.967584 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:41.967584 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:41.967758 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:41.967801 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.967803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.968520 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.968522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.969050 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:41.969053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.969399 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.969400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:41.969456 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.969931 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:41.969936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:41.971678 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
W20260924 19:37:41.971817 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:41.967331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60929 (local address 127.0.98.62:42509)
0924 19:37:41.967391 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:41.967393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:41.967450 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:37:41.967515 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:41.967516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:41.967530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:41.967578 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:41.967918 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.968449 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.969125 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:41.969339 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:41.969394 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:41.971760 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60929: BlockingRecv error: recv got EOF from 127.0.98.3:60929 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:42.756939 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:42.748819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:59331)
0924 19:37:42.748930 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:42.748940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:42.749004 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:42.749145 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:42.749146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:42.749307 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:42.749357 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.749358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.750029 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.750030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.750598 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.750600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.750913 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.750914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.750949 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.755121 (+ 4172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:42.755126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:42.756879 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
W20260924 19:37:42.757072 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:42.748930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59331 (local address 127.0.98.62:42509)
0924 19:37:42.749034 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:42.749036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:42.749044 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:42.749065 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:42.749066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:42.749075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:42.749132 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:42.749425 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.749962 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.750665 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.750860 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.750914 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.756999 (+ 6085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59331: BlockingRecv error: recv got EOF from 127.0.98.1:59331 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:42.862118 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:42.857853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:50889)
0924 19:37:42.857963 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:42.857975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:42.858045 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:42.858202 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:42.858203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:42.858377 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:42.858425 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.858427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.859077 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.859078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.859613 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.859615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.860008 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.860009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.860078 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.860517 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:42.860522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:42.862060 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
W20260924 19:37:42.862228 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:42.857913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50889 (local address 127.0.98.62:42509)
0924 19:37:42.857981 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:42.857983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:42.858024 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:37:42.858114 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:42.858115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:42.858127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:42.858182 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:42.858496 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.858999 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.859696 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.859919 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.860005 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.862166 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50889: BlockingRecv error: recv got EOF from 127.0.98.2:50889 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:42 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:42.976428 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:42.972171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:34061)
0924 19:37:42.972300 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:42.972314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:42.972380 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:42.972510 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:42.972511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:42.972696 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:42.972739 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.972741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.973377 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.973378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.973949 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:42.973952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.974261 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.974263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:42.974316 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.974715 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:42.974718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:42.976354 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
W20260924 19:37:42.976495 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:42.972259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34061 (local address 127.0.98.62:42509)
0924 19:37:42.972350 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:42.972352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:42.972361 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:42.972445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:42.972446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:42.972458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:42.972497 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:42.972811 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.973294 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.974023 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:42.974205 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:42.974258 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:42.976442 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34061: BlockingRecv error: recv got EOF from 127.0.98.3:34061 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:43.761868 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:43.757414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:47735)
0924 19:37:43.757537 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:43.757549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:43.757622 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:43.757818 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:43.757819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:43.757985 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:43.758031 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.758033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.758755 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.758756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.759345 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.759347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.759673 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.759673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.759715 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.760133 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:43.760137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:43.761793 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260924 19:37:43.761934 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:43.757483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47735 (local address 127.0.98.62:42509)
0924 19:37:43.757609 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:43.757610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:43.757619 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:43.757666 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:43.757666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:43.757676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:43.757728 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:43.758094 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.758699 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.759416 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.759615 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.759659 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.761870 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47735: BlockingRecv error: recv got EOF from 127.0.98.1:47735 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:43.866711 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:43.862567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:34531)
0924 19:37:43.862684 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:43.862693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:43.862740 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:43.862841 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:43.862842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:43.862995 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:43.863040 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.863041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.863636 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.863637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.864187 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.864188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.864506 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.864506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.864541 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.864929 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:43.864933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:43.866646 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
W20260924 19:37:43.866794 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:43.862690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34531 (local address 127.0.98.62:42509)
0924 19:37:43.862745 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:43.862746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:43.862755 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:37:43.862778 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:43.862779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:43.862788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:43.862831 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:43.863093 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.863583 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.864248 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.864454 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.864505 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.866736 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34531: BlockingRecv error: recv got EOF from 127.0.98.2:34531 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:43 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:43.981007 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:43.976851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:58269)
0924 19:37:43.976975 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:43.976985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:43.977046 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:43.977155 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:43.977155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:43.977301 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:43.977345 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.977347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.977958 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.977959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.978489 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:43.978491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.978775 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.978776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:43.978817 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.979258 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:43.979262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:43.980923 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260924 19:37:43.981060 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:43.976920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58269 (local address 127.0.98.62:42509)
0924 19:37:43.976978 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:43.976980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:43.977039 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:37:43.977096 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:43.977097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:43.977107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:43.977143 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:43.977393 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.977892 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.978547 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:43.978717 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:43.978759 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:43.981012 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58269: BlockingRecv error: recv got EOF from 127.0.98.3:58269 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:44.766341 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:44.762286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:60521)
0924 19:37:44.762419 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:44.762429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:44.762475 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:44.762596 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:44.762597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:44.762750 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:44.762799 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.762800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.763415 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.763416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.763985 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.763987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.764288 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.764289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.764323 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.764761 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:44.764765 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:44.766285 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
W20260924 19:37:44.766430 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:44.762352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60521 (local address 127.0.98.62:42509)
0924 19:37:44.762475 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:44.762477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:44.762484 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:37:44.762510 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:44.762511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:44.762523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:44.762581 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:44.762851 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.763354 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.764045 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.764238 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.764287 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.766360 (+ 2073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60521: BlockingRecv error: recv got EOF from 127.0.98.1:60521 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:44.871495 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:44.867146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:57129)
0924 19:37:44.867262 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:44.867274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:44.867339 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:44.867465 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:44.867465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:44.867655 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:44.867700 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.867702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.868386 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.868388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.868898 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.868901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.869221 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.869222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.869274 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.869678 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:44.869683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:44.871419 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
W20260924 19:37:44.871565 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:44.867232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57129 (local address 127.0.98.62:42509)
0924 19:37:44.867295 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:44.867297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:44.867320 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:37:44.867395 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:44.867395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:44.867408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:44.867452 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:44.867771 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.868319 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.868970 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.869165 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.869216 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.871514 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57129: BlockingRecv error: recv got EOF from 127.0.98.2:57129 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:44 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:44.985816 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:44.981428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:35805)
0924 19:37:44.981585 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:44.981597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:44.981661 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:44.981825 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:44.981825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:44.982011 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:44.982056 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.982059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.982690 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.982690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.983202 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:44.983203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.983531 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.983531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:44.983570 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.984067 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:44.984072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:44.985740 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260924 19:37:44.985893 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:44.981501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35805 (local address 127.0.98.62:42509)
0924 19:37:44.981560 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:44.981561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:44.981655 (+ 94us) server_negotiation.cc:407] Connection header received
0924 19:37:44.981716 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:44.981717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:44.981733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:44.981801 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:44.982130 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.982628 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.983272 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:44.983470 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:44.983529 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:44.985840 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35805: BlockingRecv error: recv got EOF from 127.0.98.3:35805 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 19:37:45 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:45 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:45.771265 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:45.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:56371)
0924 19:37:45.766938 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:45.766948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:45.766992 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:45.767180 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:45.767180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:45.767329 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:45.767375 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:45.767377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.768084 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.768085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:45.768608 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:45.768610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.768953 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.768954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:45.769000 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:45.769463 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:45.769466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:45.771189 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260924 19:37:45.771347 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:45.766914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56371 (local address 127.0.98.62:42509)
0924 19:37:45.767057 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:45.767059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:45.767069 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:37:45.767091 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:45.767091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:45.767102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:45.767171 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:45.767438 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.767976 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.768676 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.768882 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.768942 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:45.771282 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56371: BlockingRecv error: recv got EOF from 127.0.98.1:56371 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
Sep 24 19:37:45 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:37:45 dist-test-slave-6kf3 krb5kdc[8577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:37:45.876340 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:45.872022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:48473)
0924 19:37:45.872156 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:45.872167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:45.872221 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:45.872335 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:45.872336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:45.872490 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:45.872534 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:45.872535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.873206 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.873207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:45.873787 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:45.873790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.874116 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.874117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:45.874159 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:45.874535 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:45.874539 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:45.876273 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260924 19:37:45.876406 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:45.872078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48473 (local address 127.0.98.62:42509)
0924 19:37:45.872132 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:45.872134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:45.872214 (+ 80us) server_negotiation.cc:407] Connection header received
0924 19:37:45.872264 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:45.872265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:45.872275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:45.872324 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:45.872590 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.873115 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.873862 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:45.874064 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:45.874104 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:45.876355 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48473: BlockingRecv error: recv got EOF from 127.0.98.2:48473 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:37:45.913165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9160
I20260924 19:37:45.917388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9312
I20260924 19:37:45.921375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9445
I20260924 19:37:45.924978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9090
2026-09-24T19:37:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17225 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:45Z Disabled control of system clock
I20260924 19:37:45.950546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35165
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:37:46.028045 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.028246 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.028272 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.029399 9641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:46.029438 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.029454 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.029469 9641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:46.029484 9641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:46.030786 9641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.031018 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.031266 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.035573 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.035642 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.035753 9641 server_base.cc:1034] running on GCE node
W20260924 19:37:46.035645 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.036058 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.036298 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.037438 9641 hybrid_clock.cc:648] HybridClock initialized: now 1790278666037427 us; error 25 us; skew 500 ppm
I20260924 19:37:46.038569 9641 webserver.cc:460] Webserver started at http://127.0.98.62:46713/ using document root <none> and password file <none>
I20260924 19:37:46.038775 9641 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.038818 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.038970 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.039969 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "c53b8829236f4c8eb5b57f4609079108"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "73fe80c8bc6f50e1a01eb7268b74289f"
server_key_iv: "d18e73701942085701ed88103e5ed7d1"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.040298 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "c53b8829236f4c8eb5b57f4609079108"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "73fe80c8bc6f50e1a01eb7268b74289f"
server_key_iv: "d18e73701942085701ed88103e5ed7d1"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.041427 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:46.042069 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.042199 9641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:46.042271 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "c53b8829236f4c8eb5b57f4609079108"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "73fe80c8bc6f50e1a01eb7268b74289f"
server_key_iv: "d18e73701942085701ed88103e5ed7d1"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.042335 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.057904 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.058176 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.058300 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.061836 9641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35165
I20260924 19:37:46.061865 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35165 every 8 connection(s)
I20260924 19:37:46.062165 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:46.062753 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:46.064282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9641
I20260924 19:37:46.064375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:37:46.064563 392 external_mini_cluster.cc:1428] Setting key 59d4aae296457acb8a349d0ca15e02b5
I20260924 19:37:46.065538 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108: Bootstrap starting.
I20260924 19:37:46.066148 9708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:46.066406 9708 log.cc:824] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:46.067011 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108: No bootstrap required, opened a new log
I20260924 19:37:46.068825 9708 raft_consensus.cc:348] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } }
I20260924 19:37:46.068966 9708 raft_consensus.cc:374] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:46.069001 9708 raft_consensus.cc:729] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53b8829236f4c8eb5b57f4609079108, State: Initialized, Role: FOLLOWER
I20260924 19:37:46.069124 9708 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } }
I20260924 19:37:46.069206 9708 raft_consensus.cc:388] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:46.069236 9708 raft_consensus.cc:482] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:46.069301 9708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:46.069677 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.065041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.98.62:35165)
0924 19:37:46.065217 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.065225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.065249 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:37:46.065667 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.065675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.065735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.065847 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.066057 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.067061 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.067788 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.068226 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.068331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.068986 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.068990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.068990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.069178 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.069210 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.069220 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.069280 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.069364 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.069528 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260924 19:37:46.069872 9708 raft_consensus.cc:504] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } }
I20260924 19:37:46.069973 9708 leader_election.cc:302] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53b8829236f4c8eb5b57f4609079108; no voters:
I20260924 19:37:46.070101 9708 leader_election.cc:288] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:37:46.070149 9713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:46.070288 9713 raft_consensus.cc:686] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [term 1 LEADER]: Becoming Leader. State: Replica: c53b8829236f4c8eb5b57f4609079108, State: Running, Role: LEADER
I20260924 19:37:46.070331 9708 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:46.070371 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } }
I20260924 19:37:46.070675 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53b8829236f4c8eb5b57f4609079108. Latest consensus state: current_term: 1 leader_uuid: "c53b8829236f4c8eb5b57f4609079108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } } }
I20260924 19:37:46.070806 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:46.070700 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53b8829236f4c8eb5b57f4609079108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53b8829236f4c8eb5b57f4609079108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35165 } } }
I20260924 19:37:46.071017 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108 [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:46.071269 9722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:46.071698 9722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:46.073101 9722 catalog_manager.cc:1347] Generated new cluster ID: 3e9af9fa65f74b5aa13c1a87b869a0da
I20260924 19:37:46.073151 9722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:37:46.079880 9722 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:37:46.080350 9722 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:46.082327 9722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c53b8829236f4c8eb5b57f4609079108: Generated new TSK 0
I20260924 19:37:46.082506 9722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:37:46.087816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--builtin_ntp_servers=127.0.98.20:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:37:46.183465 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.183651 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.183676 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.184814 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.184851 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.184875 9732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:46.186162 9732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.186362 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.186625 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.191150 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.191239 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.191272 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.191527 9732 server_base.cc:1034] running on GCE node
I20260924 19:37:46.191710 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.191953 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.193099 9732 hybrid_clock.cc:648] HybridClock initialized: now 1790278666193087 us; error 25 us; skew 500 ppm
I20260924 19:37:46.194358 9732 webserver.cc:460] Webserver started at http://127.0.98.1:42773/ using document root <none> and password file <none>
I20260924 19:37:46.194588 9732 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.194630 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.194737 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.195813 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "637d084389f54087a9ee67be751c0cb4"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "77d1e48cc767f26e9d2bcace60b2cbd3"
server_key_iv: "2f06591bb3940a104415867e3f412a6f"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.196178 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "637d084389f54087a9ee67be751c0cb4"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "77d1e48cc767f26e9d2bcace60b2cbd3"
server_key_iv: "2f06591bb3940a104415867e3f412a6f"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.197360 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:46.198053 9746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.198199 9732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:46.198278 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "637d084389f54087a9ee67be751c0cb4"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "77d1e48cc767f26e9d2bcace60b2cbd3"
server_key_iv: "2f06591bb3940a104415867e3f412a6f"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.198347 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.208798 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.209064 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.209178 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.209364 9732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.209705 9732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.209740 9732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.209761 9732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.209784 9732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.215377 9732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35963
I20260924 19:37:46.215423 9859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35963 every 8 connection(s)
I20260924 19:37:46.215708 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:37:46.220559 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.216384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42259 (local address 127.0.98.62:35165)
0924 19:37:46.216457 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.216459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.216640 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:37:46.217174 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.217175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.217186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.217236 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.217855 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.218326 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.219233 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.219439 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.219491 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.220314 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.220315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.220315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.220320 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.220334 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.220338 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.220381 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.220461 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.220484 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:37:46.220767 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.216377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35165 (local address 127.0.98.1:42259)
0924 19:37:46.216534 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.216564 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.217079 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.217300 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.217303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.217311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.217777 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.217782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.218403 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.218404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.219158 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.219160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.219503 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.219504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.219615 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.220193 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.220193 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.220220 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.220239 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.220392 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.220396 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.220399 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.220459 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.220555 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260924 19:37:46.221213 9860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35165
I20260924 19:37:46.221287 9860 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.221434 9860 heartbeater.cc:511] Master 127.0.98.62:35165 requested a full tablet report, sending...
I20260924 19:37:46.221843 9672 ts_manager.cc:194] Registered new tserver with Master: 637d084389f54087a9ee67be751c0cb4 (127.0.98.1:35963)
I20260924 19:37:46.222426 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42259
I20260924 19:37:46.222775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9732
I20260924 19:37:46.222854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:37:46.223037 392 external_mini_cluster.cc:1428] Setting key 5dfbcea6ed4dd844b701e0e44a98e1f9
I20260924 19:37:46.223783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--builtin_ntp_servers=127.0.98.20:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:37:46.302646 9863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.302831 9863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.302857 9863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.303990 9863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.304024 9863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.304055 9863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:46.305325 9863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.305534 9863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.305788 9863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.310520 9868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.310520 9871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.310672 9869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.310849 9863 server_base.cc:1034] running on GCE node
I20260924 19:37:46.311046 9863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.311225 9863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.312353 9863 hybrid_clock.cc:648] HybridClock initialized: now 1790278666312341 us; error 27 us; skew 500 ppm
I20260924 19:37:46.313452 9863 webserver.cc:460] Webserver started at http://127.0.98.2:37521/ using document root <none> and password file <none>
I20260924 19:37:46.313643 9863 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.313692 9863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.313807 9863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.314824 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "9d53683b98c14b1e86a97467f12497f3"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "de36e338c7745cbafe81af40e6b455b7"
server_key_iv: "06b6d618b49a62af4fa06f9549195198"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.315145 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9d53683b98c14b1e86a97467f12497f3"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "de36e338c7745cbafe81af40e6b455b7"
server_key_iv: "06b6d618b49a62af4fa06f9549195198"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.316340 9863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:46.316915 9877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.317042 9863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:46.317117 9863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "9d53683b98c14b1e86a97467f12497f3"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "de36e338c7745cbafe81af40e6b455b7"
server_key_iv: "06b6d618b49a62af4fa06f9549195198"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.317174 9863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.326833 9863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.327071 9863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.327188 9863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.327375 9863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.327646 9863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.327677 9863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.327700 9863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.327718 9863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.332945 9863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39697
I20260924 19:37:46.332980 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39697 every 8 connection(s)
I20260924 19:37:46.333244 9863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:37:46.337940 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.333848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43669 (local address 127.0.98.62:35165)
0924 19:37:46.333903 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.333904 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.334113 (+ 209us) server_negotiation.cc:407] Connection header received
0924 19:37:46.334601 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.334602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.334614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.334656 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.335257 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.335740 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.336710 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.336908 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.336962 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.337693 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.337694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.337695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.337699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.337714 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.337717 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.337762 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.337839 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.337863 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:37:46.338109 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.333825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35165 (local address 127.0.98.2:43669)
0924 19:37:46.334002 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.334037 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.334497 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.334731 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.334734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.334742 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.335181 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.335185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.335820 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.335821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.336626 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.336629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.336972 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.336973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.337100 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.337551 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.337552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.337579 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.337601 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.337767 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.337771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.337774 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.337833 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.337928 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260924 19:37:46.338567 9991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35165
I20260924 19:37:46.338636 9991 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.338770 9991 heartbeater.cc:511] Master 127.0.98.62:35165 requested a full tablet report, sending...
I20260924 19:37:46.339094 9672 ts_manager.cc:194] Registered new tserver with Master: 9d53683b98c14b1e86a97467f12497f3 (127.0.98.2:39697)
I20260924 19:37:46.339414 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43669
I20260924 19:37:46.339743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9863
I20260924 19:37:46.339833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:37:46.340040 392 external_mini_cluster.cc:1428] Setting key f41cc912ed5e7690d4ab856acc9e7f9d
I20260924 19:37:46.340793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--builtin_ntp_servers=127.0.98.20:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:37:46.416826 9994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.417013 9994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.417032 9994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.418146 9994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.418183 9994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.418206 9994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:46.419493 9994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.419692 9994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.419973 9994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.424391 9999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.424512 10002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.424389 10000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.424759 9994 server_base.cc:1034] running on GCE node
I20260924 19:37:46.424958 9994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.425170 9994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.426318 9994 hybrid_clock.cc:648] HybridClock initialized: now 1790278666426302 us; error 31 us; skew 500 ppm
I20260924 19:37:46.427551 9994 webserver.cc:460] Webserver started at http://127.0.98.3:34255/ using document root <none> and password file <none>
I20260924 19:37:46.427769 9994 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.427815 9994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.428035 9994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.429104 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b8660c070254c01afb84d0e4278d294"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "8957fd0e9dd635b996bf854700b311a8"
server_key_iv: "debcc27b983ba0dbaf6c2a21cf0a7482"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.429447 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b8660c070254c01afb84d0e4278d294"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "8957fd0e9dd635b996bf854700b311a8"
server_key_iv: "debcc27b983ba0dbaf6c2a21cf0a7482"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.430687 9994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:46.431386 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.431514 9994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:46.431583 9994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "3b8660c070254c01afb84d0e4278d294"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "8957fd0e9dd635b996bf854700b311a8"
server_key_iv: "debcc27b983ba0dbaf6c2a21cf0a7482"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.431648 9994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.444723 9994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.444975 9994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.445103 9994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.445297 9994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.445585 9994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.445616 9994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.445643 9994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.445660 9994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.450883 9994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40959
I20260924 19:37:46.450929 10121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40959 every 8 connection(s)
I20260924 19:37:46.451203 9994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:37:46.454602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 9994
I20260924 19:37:46.454686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:37:46.454883 392 external_mini_cluster.cc:1428] Setting key a37dd724b7fc1f93bc95af6d2a993b82
I20260924 19:37:46.456084 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.451801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44199 (local address 127.0.98.62:35165)
0924 19:37:46.451847 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.451849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.452155 (+ 306us) server_negotiation.cc:407] Connection header received
0924 19:37:46.452603 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.452604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.452612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.452660 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.453244 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.453724 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.454820 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.455041 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.455103 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.455866 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.455867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.455868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.455871 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.455885 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.455888 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.455924 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.456005 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.456029 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:37:46.456254 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.451791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35165 (local address 127.0.98.3:44199)
0924 19:37:46.452045 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.452077 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.452530 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.452721 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.452724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.452731 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.453175 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.453179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.453803 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.453804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.454748 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:46.454751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:46.455108 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:46.455109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:46.455263 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:46.455754 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.455754 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.455782 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.455804 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.455926 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.455947 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.455950 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.456005 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.456077 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260924 19:37:46.456650 10122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35165
I20260924 19:37:46.456723 10122 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.456882 10122 heartbeater.cc:511] Master 127.0.98.62:35165 requested a full tablet report, sending...
I20260924 19:37:46.457186 9672 ts_manager.cc:194] Registered new tserver with Master: 3b8660c070254c01afb84d0e4278d294 (127.0.98.3:40959)
I20260924 19:37:46.457533 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44199
I20260924 19:37:46.466372 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:37:46.467970 9711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:46.467532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.0.98.62:35165)
0924 19:37:46.467569 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.467571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.467666 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:37:46.467714 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.467714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.467723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.467766 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.467883 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49896: BlockingRecv error: recv got EOF from 127.0.0.1:49896 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:37:46.468046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9732
I20260924 19:37:46.471817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9863
I20260924 19:37:46.475307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9994
I20260924 19:37:46.478645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 9641
2026-09-24T19:37:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (552 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:46Z Disabled control of system clock
I20260924 19:37:46.502108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33107
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:37:46.578415 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.578608 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.578627 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.579725 10133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:46.579763 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.579777 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.579792 10133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:46.579815 10133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:46.581132 10133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.581347 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.581605 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.586067 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.586062 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.586105 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.586090 10133 server_base.cc:1034] running on GCE node
I20260924 19:37:46.586432 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.586675 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.587814 10133 hybrid_clock.cc:648] HybridClock initialized: now 1790278666587800 us; error 29 us; skew 500 ppm
I20260924 19:37:46.588992 10133 webserver.cc:460] Webserver started at http://127.0.98.62:33447/ using document root <none> and password file <none>
I20260924 19:37:46.589184 10133 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.589229 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.589342 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.590313 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "24b5e6012b6a40ac933bedaccc33793a"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "b5e19b30cba52c172b8885a9b47ccbf2"
server_key_iv: "9632e773e88293091ccde7edb464fe9b"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.590624 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "24b5e6012b6a40ac933bedaccc33793a"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "b5e19b30cba52c172b8885a9b47ccbf2"
server_key_iv: "9632e773e88293091ccde7edb464fe9b"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.591742 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:46.592456 10147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.592592 10133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.592667 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "24b5e6012b6a40ac933bedaccc33793a"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "b5e19b30cba52c172b8885a9b47ccbf2"
server_key_iv: "9632e773e88293091ccde7edb464fe9b"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.592729 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.595849 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.596110 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.596232 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.599704 10133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33107
I20260924 19:37:46.599730 10199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33107 every 8 connection(s)
I20260924 19:37:46.600064 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:46.600625 10200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:46.603067 10200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a: Bootstrap starting.
I20260924 19:37:46.603583 10200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:46.603837 10200 log.cc:824] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:46.604413 10200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a: No bootstrap required, opened a new log
I20260924 19:37:46.605543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10133
I20260924 19:37:46.605635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:37:46.605804 392 external_mini_cluster.cc:1428] Setting key 9fcbb11ae18f063d01a2af839e56e1d8
I20260924 19:37:46.605957 10200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260924 19:37:46.606082 10200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:46.606114 10200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b5e6012b6a40ac933bedaccc33793a, State: Initialized, Role: FOLLOWER
I20260924 19:37:46.606259 10200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260924 19:37:46.606343 10200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:46.606371 10200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:46.606416 10200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:46.606884 10200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260924 19:37:46.606987 10200 leader_election.cc:302] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b5e6012b6a40ac933bedaccc33793a; no voters:
I20260924 19:37:46.607120 10200 leader_election.cc:288] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:37:46.607162 10205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:46.607254 10205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [term 1 LEADER]: Becoming Leader. State: Replica: 24b5e6012b6a40ac933bedaccc33793a, State: Running, Role: LEADER
I20260924 19:37:46.607318 10205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260924 19:37:46.607342 10200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:46.607573 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b5e6012b6a40ac933bedaccc33793a. Latest consensus state: current_term: 1 leader_uuid: "24b5e6012b6a40ac933bedaccc33793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } } }
I20260924 19:37:46.607631 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:46.607849 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b5e6012b6a40ac933bedaccc33793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b5e6012b6a40ac933bedaccc33793a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } } }
I20260924 19:37:46.607967 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:46.608153 10209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:46.608300 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.606287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.0.98.62:33107)
0924 19:37:46.606544 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.606553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.606578 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:37:46.606671 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.606676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.606708 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.606818 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.607497 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.607501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.607501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.607506 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.607547 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.607558 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.607618 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.607778 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.607841 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260924 19:37:46.608647 10209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:46.610101 10209 catalog_manager.cc:1347] Generated new cluster ID: b9507c93159e485cb2f0ba53003078ed
I20260924 19:37:46.610157 10209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:37:46.628393 10209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:37:46.628572 10209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:46.635834 10209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b5e6012b6a40ac933bedaccc33793a: Generated new TSK 0
I20260924 19:37:46.636030 10209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:37:46.638343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--builtin_ntp_servers=127.0.98.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:37:46.720909 10224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.721104 10224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.721134 10224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.722254 10224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.722294 10224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.722328 10224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:46.723706 10224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.723990 10224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.724268 10224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.728976 10230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.729038 10229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.729053 10232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.729485 10224 server_base.cc:1034] running on GCE node
I20260924 19:37:46.729691 10224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.729944 10224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.731104 10224 hybrid_clock.cc:648] HybridClock initialized: now 1790278666731091 us; error 25 us; skew 500 ppm
I20260924 19:37:46.732489 10224 webserver.cc:460] Webserver started at http://127.0.98.1:39425/ using document root <none> and password file <none>
I20260924 19:37:46.732788 10224 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.732849 10224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.732971 10224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.734370 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9dd8af65fac4e9793b1757f37b994b2"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "fb4f46c4b161895f32fc243169946481"
server_key_iv: "12272bfe8d1c0c3c2a051efdc6385eec"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.734782 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9dd8af65fac4e9793b1757f37b994b2"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "fb4f46c4b161895f32fc243169946481"
server_key_iv: "12272bfe8d1c0c3c2a051efdc6385eec"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.736310 10224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 19:37:46.737200 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.737367 10224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:46.737449 10224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "c9dd8af65fac4e9793b1757f37b994b2"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "fb4f46c4b161895f32fc243169946481"
server_key_iv: "12272bfe8d1c0c3c2a051efdc6385eec"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.737512 10224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.741950 10224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.742215 10224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.742327 10224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.742511 10224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.742807 10224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.742844 10224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.742866 10224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.742887 10224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.748445 10224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39327
I20260924 19:37:46.748519 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39327 every 8 connection(s)
I20260924 19:37:46.748772 10224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:37:46.751448 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.749393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36421 (local address 127.0.98.62:33107)
0924 19:37:46.749446 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.749448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.749690 (+ 242us) server_negotiation.cc:407] Connection header received
0924 19:37:46.750144 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.750145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.750152 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.750209 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.751217 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.751218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.751219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.751223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.751237 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.751241 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.751291 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.751365 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.751381 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:37:46.751614 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.749397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.1:36421)
0924 19:37:46.749565 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.749603 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.750047 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.750277 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.750280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.750288 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.751074 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.751075 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.751104 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.751129 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.751296 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.751300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.751303 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.751368 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.751426 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260924 19:37:46.752054 10352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260924 19:37:46.752135 10352 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.752259 10352 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
I20260924 19:37:46.752677 10164 ts_manager.cc:194] Registered new tserver with Master: c9dd8af65fac4e9793b1757f37b994b2 (127.0.98.1:39327)
I20260924 19:37:46.752933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10224
I20260924 19:37:46.753026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:37:46.753191 392 external_mini_cluster.cc:1428] Setting key d1656cee9b4ba37518d60e1b43be4eab
I20260924 19:37:46.754050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--builtin_ntp_servers=127.0.98.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:37:46.832960 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.833155 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.833177 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.834318 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.834362 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.834447 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:46.835994 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.836205 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.836478 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.841150 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.841225 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.841331 10355 server_base.cc:1034] running on GCE node
W20260924 19:37:46.841444 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.841647 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.841838 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.842963 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790278666842954 us; error 25 us; skew 500 ppm
I20260924 19:37:46.844098 10355 webserver.cc:460] Webserver started at http://127.0.98.2:43827/ using document root <none> and password file <none>
I20260924 19:37:46.844295 10355 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.844337 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.844456 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.845582 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b2c827e1a584da8922aee52bc0eb388"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "5aa93ece116a5d1e483b893286727d8a"
server_key_iv: "361c13e933ede2493a88cf7f7409ddaf"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.845934 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b2c827e1a584da8922aee52bc0eb388"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "5aa93ece116a5d1e483b893286727d8a"
server_key_iv: "361c13e933ede2493a88cf7f7409ddaf"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.847213 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:37:46.847819 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.847975 10355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:46.848067 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "2b2c827e1a584da8922aee52bc0eb388"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "5aa93ece116a5d1e483b893286727d8a"
server_key_iv: "361c13e933ede2493a88cf7f7409ddaf"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.848125 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.851229 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.851450 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.851569 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.851754 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.852047 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.852080 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.852104 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.852128 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.857249 10355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45391
I20260924 19:37:46.857296 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45391 every 8 connection(s)
I20260924 19:37:46.857575 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:37:46.858316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10355
I20260924 19:37:46.858417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:37:46.858597 392 external_mini_cluster.cc:1428] Setting key 708314e43b4077346211a318ac5857a0
I20260924 19:37:46.859473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--builtin_ntp_servers=127.0.98.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:37:46.860700 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.858262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33469 (local address 127.0.98.62:33107)
0924 19:37:46.858309 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.858311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.858585 (+ 274us) server_negotiation.cc:407] Connection header received
0924 19:37:46.859139 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.859140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.859145 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.859185 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.860417 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.860418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.860419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.860423 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.860439 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.860443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.860493 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.860632 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.860648 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:37:46.860889 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.858268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.2:33469)
0924 19:37:46.858456 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.858498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.859056 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.859254 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.859258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.859267 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.860285 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.860285 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.860313 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.860339 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.860518 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.860524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.860537 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.860631 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.860692 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260924 19:37:46.861340 10483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260924 19:37:46.861423 10483 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.861552 10483 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
I20260924 19:37:46.861876 10164 ts_manager.cc:194] Registered new tserver with Master: 2b2c827e1a584da8922aee52bc0eb388 (127.0.98.2:45391)
W20260924 19:37:46.936691 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:46.936880 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:46.936899 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:46.937999 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:46.938031 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:46.938054 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:46.939293 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:46.939462 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:46.939688 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:46.944355 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.944398 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:46.944424 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:46.944648 10486 server_base.cc:1034] running on GCE node
I20260924 19:37:46.944814 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:46.945019 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:46.946170 10486 hybrid_clock.cc:648] HybridClock initialized: now 1790278666946151 us; error 35 us; skew 500 ppm
I20260924 19:37:46.947307 10486 webserver.cc:460] Webserver started at http://127.0.98.3:42553/ using document root <none> and password file <none>
I20260924 19:37:46.947503 10486 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:46.947551 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:46.947669 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:46.948696 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a65d54f5295423d83fb7053a2cb9df8"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "18f7a199d6a760ca10e3d58778ec2c16"
server_key_iv: "0ea7fc919dec3a3d2903e27cc0a8192d"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.948998 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a65d54f5295423d83fb7053a2cb9df8"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "18f7a199d6a760ca10e3d58778ec2c16"
server_key_iv: "0ea7fc919dec3a3d2903e27cc0a8192d"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.950157 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:37:46.950831 10500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.950989 10486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:46.951081 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "5a65d54f5295423d83fb7053a2cb9df8"
format_stamp: "Formatted at 2026-09-24 19:37:46 on dist-test-slave-6kf3"
server_key: "18f7a199d6a760ca10e3d58778ec2c16"
server_key_iv: "0ea7fc919dec3a3d2903e27cc0a8192d"
server_key_version: "encryptionkey@0"
I20260924 19:37:46.951135 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:46.954255 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:46.954474 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:46.954582 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:46.954766 10486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:46.955039 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:46.955070 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.955094 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:46.955117 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:46.960299 10486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41527
I20260924 19:37:46.960338 10613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41527 every 8 connection(s)
I20260924 19:37:46.960602 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:37:46.963151 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.961208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39535 (local address 127.0.98.62:33107)
0924 19:37:46.961256 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.961258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.961457 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:37:46.961846 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.961847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.961852 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.961917 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.962965 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:46.962966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:46.962966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:37:46.962970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:46.962984 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:37:46.962988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:46.963037 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:46.963087 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:37:46.963098 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:37:46.963308 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:46.961192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.3:39535)
0924 19:37:46.961347 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:46.961381 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:46.961780 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:46.961967 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:46.961969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:46.961976 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:46.962804 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:37:46.962805 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:46.962830 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:37:46.962856 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:46.963043 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:46.963045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:46.963047 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:37:46.963093 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:37:46.963133 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260924 19:37:46.963732 10614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260924 19:37:46.963747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10486
I20260924 19:37:46.963814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:37:46.963811 10614 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:46.963974 10614 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
I20260924 19:37:46.964021 392 external_mini_cluster.cc:1428] Setting key 32dd8bb3fc8d4ae03ac9ffad52c6063c
I20260924 19:37:46.964319 10164 ts_manager.cc:194] Registered new tserver with Master: 5a65d54f5295423d83fb7053a2cb9df8 (127.0.98.3:41527)
I20260924 19:37:46.964795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:37:46.966281 10204 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:37:46.965939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.62:33107)
0924 19:37:46.965973 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:46.965974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:46.966070 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:37:46.966117 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:46.966118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:46.966121 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:46.966162 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:46.966240 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34482: BlockingRecv error: recv got EOF from 127.0.0.1:34482 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:37:46.966382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10224
I20260924 19:37:46.970724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10355
I20260924 19:37:46.974560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10486
I20260924 19:37:46.978163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10133
2026-09-24T19:37:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (499 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:37:47 dist-test-slave-6kf3 krb5kdc[10624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:37:47 dist-test-slave-6kf3 krb5kdc[10624](info): set up 2 sockets
Sep 24 19:37:47 dist-test-slave-6kf3 krb5kdc[10624](info): commencing operation
krb5kdc: starting...
W20260924 19:37:47.923959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:37:47 dist-test-slave-6kf3 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:37:48.026583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:48.106124 10637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:48.106319 10637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:48.106340 10637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:48.107415 10637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:37:48.107450 10637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:48.107462 10637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:48.107473 10637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:37:48.107486 10637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:37:48.108887 10637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:48.109082 10637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:48.109251 10637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:48.113629 10645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.113664 10643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.113801 10642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:48.113945 10637 server_base.cc:1034] running on GCE node
I20260924 19:37:48.114173 10637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:48.114452 10637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:48.115610 10637 hybrid_clock.cc:648] HybridClock initialized: now 1790278668115590 us; error 31 us; skew 500 ppm
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:48.121728 10637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:37:48.122152 10637 webserver.cc:460] Webserver started at http://127.0.98.62:38647/ using document root <none> and password file <none>
I20260924 19:37:48.122344 10637 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:48.122390 10637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:48.122516 10637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:48.123525 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "22acaf1570c740b7b2fcce73205e49df"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "7df4937480e42e65a7cb67ed9eee4b93"
server_key_iv: "e9a548b11d0c0f1e3d947a25a9d807dc"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.123842 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "22acaf1570c740b7b2fcce73205e49df"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "7df4937480e42e65a7cb67ed9eee4b93"
server_key_iv: "e9a548b11d0c0f1e3d947a25a9d807dc"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.125026 10637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:37:48.125629 10652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.125770 10637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:48.125849 10637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "22acaf1570c740b7b2fcce73205e49df"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "7df4937480e42e65a7cb67ed9eee4b93"
server_key_iv: "e9a548b11d0c0f1e3d947a25a9d807dc"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.125905 10637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:48.132817 10637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:48.134362 10637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:48.134492 10637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:48.138054 10637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42099
I20260924 19:37:48.138090 10704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42099 every 8 connection(s)
I20260924 19:37:48.138373 10637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:37:48.139218 10705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:48.140405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10637
I20260924 19:37:48.140491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:37:48.140650 392 external_mini_cluster.cc:1428] Setting key 57deb95eaace044f8de14dc7b4c461b9
I20260924 19:37:48.141583 10705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: Bootstrap starting.
I20260924 19:37:48.142184 10705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:48.142483 10705 log.cc:824] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:48.143092 10705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: No bootstrap required, opened a new log
I20260924 19:37:48.145180 10705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } }
I20260924 19:37:48.145319 10705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:48.145350 10705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22acaf1570c740b7b2fcce73205e49df, State: Initialized, Role: FOLLOWER
I20260924 19:37:48.145473 10705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } }
I20260924 19:37:48.145556 10705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:37:48.145596 10705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:37:48.145641 10705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:48.146099 10705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } }
I20260924 19:37:48.146188 10705 leader_election.cc:302] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22acaf1570c740b7b2fcce73205e49df; no voters:
I20260924 19:37:48.146325 10705 leader_election.cc:288] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:37:48.146364 10710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:48.146508 10710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [term 1 LEADER]: Becoming Leader. State: Replica: 22acaf1570c740b7b2fcce73205e49df, State: Running, Role: LEADER
I20260924 19:37:48.146564 10705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:37:48.146651 10710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } }
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:48.146991 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22acaf1570c740b7b2fcce73205e49df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } } }
I20260924 19:37:48.147080 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [sys.catalog]: This master's current role is: LEADER
I20260924 19:37:48.146998 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22acaf1570c740b7b2fcce73205e49df. Latest consensus state: current_term: 1 leader_uuid: "22acaf1570c740b7b2fcce73205e49df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22acaf1570c740b7b2fcce73205e49df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42099 } } }
I20260924 19:37:48.147146 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df [sys.catalog]: This master's current role is: LEADER
W20260924 19:37:48.148219 10725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:37:48.148273 10725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:37:48.148315 10718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:37:48.148460 10718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:37:48.149700 10718 catalog_manager.cc:1347] Generated new cluster ID: f897eb9b0f6441bd869a15385c8b189b
I20260924 19:37:48.149745 10718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:37:48.151163 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.141102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.62:42099)
0924 19:37:48.141273 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.141283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.141306 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:37:48.141715 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.141721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.141767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.141880 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:48.142591 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.143504 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.144509 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.144887 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.144997 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.147407 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:48.147411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:48.147411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:48.147422 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:48.149253 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.149633 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.149635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.149636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.149735 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.149940 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.149940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.149941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.150234 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:48.150363 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.150821 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.150951 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260924 19:37:48.169799 10718 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:37:48.170324 10718 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:37:48.174284 10718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: Generated new TSK 0
I20260924 19:37:48.174466 10718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:37:48.200062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:48.278990 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:48.279181 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:48.279211 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:48.280334 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:48.280372 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:48.280407 10731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:37:48.281804 10731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:48.282032 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:48.282231 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:48.287004 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.286974 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.286985 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:48.287355 10731 server_base.cc:1034] running on GCE node
I20260924 19:37:48.287557 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:48.287799 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:48.289014 10731 hybrid_clock.cc:648] HybridClock initialized: now 1790278668289002 us; error 24 us; skew 500 ppm
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:48.291293 10731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:37:48.291742 10731 webserver.cc:460] Webserver started at http://127.0.98.1:39965/ using document root <none> and password file <none>
I20260924 19:37:48.291985 10731 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:48.292034 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:48.292156 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:48.293174 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b7a3d50875d41c1972bc124b66b7a03"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "223bef45fc942478bfaac8cf9910ea18"
server_key_iv: "9b8c17129d48c10da4d5ef212513fd52"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.293494 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b7a3d50875d41c1972bc124b66b7a03"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "223bef45fc942478bfaac8cf9910ea18"
server_key_iv: "9b8c17129d48c10da4d5ef212513fd52"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.294623 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:37:48.295302 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.295439 10731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:37:48.295517 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "0b7a3d50875d41c1972bc124b66b7a03"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "223bef45fc942478bfaac8cf9910ea18"
server_key_iv: "9b8c17129d48c10da4d5ef212513fd52"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.295579 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:48.318328 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:48.320080 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:48.320214 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:48.320423 10731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:48.320725 10731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:48.320757 10731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.320785 10731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:48.320808 10731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.326323 10731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34711
I20260924 19:37:48.326383 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34711 every 8 connection(s)
I20260924 19:37:48.326658 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:48.334609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10731
I20260924 19:37:48.334686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:37:48.334860 392 external_mini_cluster.cc:1428] Setting key 0811c56fd6be0e529580e2e5b33ac032
I20260924 19:37:48.335078 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.327297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49747 (local address 127.0.98.62:42099)
0924 19:37:48.327347 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.327349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.327541 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:37:48.328070 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.328070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.328079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.328131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:48.329062 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.329618 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.330534 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.330724 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.330767 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.332681 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:48.332682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:48.332683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:48.332689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:48.333917 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.334234 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.334235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.334235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.334279 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.334444 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.334444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.334444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.334554 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:48.334608 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.334907 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.334989 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:37:48.335253 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.327264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42099 (local address 127.0.98.1:49747)
0924 19:37:48.327420 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.327460 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.327985 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.328203 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.328206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.328761 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:48.328974 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.328978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.329689 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.329690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.330474 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.330476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.330777 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.330777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.330872 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.331299 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:48.331307 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:48.332601 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:48.333983 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.333987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.333989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.334183 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.334323 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.334323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.334323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.334405 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.334620 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:48.334626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:48.334790 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.334878 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.335045 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260924 19:37:48.335678 10860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42099
I20260924 19:37:48.335773 10860 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:48.335963 10860 heartbeater.cc:511] Master 127.0.98.62:42099 requested a full tablet report, sending...
I20260924 19:37:48.336440 10669 ts_manager.cc:194] Registered new tserver with Master: 0b7a3d50875d41c1972bc124b66b7a03 (127.0.98.1:34711)
I20260924 19:37:48.337042 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49747
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:37:48.356869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:48.433580 10865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:48.433772 10865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:48.433804 10865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:48.434897 10865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:48.434937 10865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:48.434971 10865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:37:48.436371 10865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:48.436605 10865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:48.436793 10865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:48.441481 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.441424 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.441459 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:48.441640 10865 server_base.cc:1034] running on GCE node
I20260924 19:37:48.441901 10865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:48.442150 10865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:48.443301 10865 hybrid_clock.cc:648] HybridClock initialized: now 1790278668443290 us; error 27 us; skew 500 ppm
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:48.445895 10865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:37:48.446375 10865 webserver.cc:460] Webserver started at http://127.0.98.2:36227/ using document root <none> and password file <none>
I20260924 19:37:48.446583 10865 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:48.446631 10865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:48.446750 10865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:48.447815 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b83ea9c696a494a8f8e141ef27b7d3a"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "de7b23b95f772321870e4fa76bdd26a0"
server_key_iv: "5acf694b3d96157d62683340a4d5ad7b"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.448195 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b83ea9c696a494a8f8e141ef27b7d3a"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "de7b23b95f772321870e4fa76bdd26a0"
server_key_iv: "5acf694b3d96157d62683340a4d5ad7b"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.449402 10865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:48.450109 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.450302 10865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:48.450376 10865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "8b83ea9c696a494a8f8e141ef27b7d3a"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "de7b23b95f772321870e4fa76bdd26a0"
server_key_iv: "5acf694b3d96157d62683340a4d5ad7b"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.450438 10865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:48.465142 10865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:48.467132 10865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:48.467255 10865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:48.467449 10865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:48.467726 10865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:48.467757 10865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.467784 10865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:48.467803 10865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.473088 10865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35573
I20260924 19:37:48.473146 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35573 every 8 connection(s)
I20260924 19:37:48.473412 10865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:48.480702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10865
I20260924 19:37:48.480794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:37:48.480973 392 external_mini_cluster.cc:1428] Setting key f4510993755d090bad24658d41f70c8a
I20260924 19:37:48.481654 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.474024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55941 (local address 127.0.98.62:42099)
0924 19:37:48.474065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.474067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.474262 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:37:48.474712 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.474713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.474722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.474777 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:48.475677 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.476160 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.477070 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.477241 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.477285 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.479203 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:48.479205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:48.479205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:48.479212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:48.480382 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.480737 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.480738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.480739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.480799 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.481049 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.481051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.481051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.481172 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:48.481234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.481486 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.481569 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:37:48.481810 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.473991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42099 (local address 127.0.98.2:55941)
0924 19:37:48.474145 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.474185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.474640 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.474845 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.474848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.475410 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:48.475611 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.475616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.476224 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.476224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.477010 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.477012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.477289 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.477289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.477379 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.477799 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:48.477807 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:48.479118 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:48.480464 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.480469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.480471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.480671 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.480874 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.480875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.480876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.480981 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.481230 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:48.481236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:48.481356 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.481448 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.481592 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260924 19:37:48.482286 10994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42099
I20260924 19:37:48.482370 10994 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:48.482547 10994 heartbeater.cc:511] Master 127.0.98.62:42099 requested a full tablet report, sending...
I20260924 19:37:48.482909 10669 ts_manager.cc:194] Registered new tserver with Master: 8b83ea9c696a494a8f8e141ef27b7d3a (127.0.98.2:35573)
I20260924 19:37:48.483263 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55941
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:37:48.502039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:37:48.578715 10999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:37:48.578910 10999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:37:48.578939 10999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:37:48.580080 10999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:37:48.580121 10999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:37:48.580156 10999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:37:48.581542 10999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:37:48.581758 10999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:37:48.581954 10999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:37:48.586493 11007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.586521 11004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:37:48.586522 11005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:37:48.586616 10999 server_base.cc:1034] running on GCE node
I20260924 19:37:48.586867 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:37:48.587039 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:37:48.588172 10999 hybrid_clock.cc:648] HybridClock initialized: now 1790278668588160 us; error 26 us; skew 500 ppm
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:37:48.590391 10999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:37:48.590790 10999 webserver.cc:460] Webserver started at http://127.0.98.3:42973/ using document root <none> and password file <none>
I20260924 19:37:48.590973 10999 fs_manager.cc:359] Metadata directory not provided
I20260924 19:37:48.591017 10999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:37:48.591140 10999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:37:48.592140 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "54945acd51644d6993b9dcfd98f57f24"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "6ea3f60c46516de44a21dac7c1f53f62"
server_key_iv: "3c57a9fe3a566461f4f8300c879e24f8"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.592445 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54945acd51644d6993b9dcfd98f57f24"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "6ea3f60c46516de44a21dac7c1f53f62"
server_key_iv: "3c57a9fe3a566461f4f8300c879e24f8"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.593626 10999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:37:48.594251 11014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.594381 10999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:37:48.594456 10999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "54945acd51644d6993b9dcfd98f57f24"
format_stamp: "Formatted at 2026-09-24 19:37:48 on dist-test-slave-6kf3"
server_key: "6ea3f60c46516de44a21dac7c1f53f62"
server_key_iv: "3c57a9fe3a566461f4f8300c879e24f8"
server_key_version: "encryptionkey@0"
I20260924 19:37:48.594511 10999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:37:48.609042 10999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:37:48.611320 10999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:37:48.611462 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:37:48.611670 10999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:37:48.611964 10999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:37:48.612003 10999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.612031 10999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:37:48.612048 10999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:37:48.617249 10999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46819
I20260924 19:37:48.617296 11127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46819 every 8 connection(s)
I20260924 19:37:48.617578 10999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:37:48.625715 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.618232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50923 (local address 127.0.98.62:42099)
0924 19:37:48.618293 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.618295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.618498 (+ 203us) server_negotiation.cc:407] Connection header received
0924 19:37:48.618967 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.618968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.618979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.619023 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:48.619860 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.620394 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.621311 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.621496 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.621538 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.623482 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:48.623484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:48.623484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:48.623491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:48.624632 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.625013 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.625014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.625014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.625056 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.625232 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.625232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.625233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.625321 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:48.625363 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.625564 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.625635 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:37:48.625833 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.618215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42099 (local address 127.0.98.3:50923)
0924 19:37:48.618374 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.618425 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.618874 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.619077 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.619081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.619605 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:37:48.619794 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.619799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.620462 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.620463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.621249 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.621251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.621580 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.621581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.621710 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.622128 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:37:48.622136 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:37:48.623383 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:37:48.624697 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.624703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.624704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.624919 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.625099 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:37:48.625099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:37:48.625100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:37:48.625193 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:37:48.625362 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:37:48.625364 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:37:48.625471 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.625545 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.625656 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260924 19:37:48.626118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 10999
I20260924 19:37:48.626219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:37:48.626291 11128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42099
I20260924 19:37:48.626372 11128 heartbeater.cc:464] Registering TS with master...
I20260924 19:37:48.626411 392 external_mini_cluster.cc:1428] Setting key 4489dc266c7b47ce600bf0edebdf1548
I20260924 19:37:48.626518 11128 heartbeater.cc:511] Master 127.0.98.62:42099 requested a full tablet report, sending...
I20260924 19:37:48.626889 10669 ts_manager.cc:194] Registered new tserver with Master: 54945acd51644d6993b9dcfd98f57f24 (127.0.98.3:46819)
I20260924 19:37:48.627252 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:37:48.627319 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50923
I20260924 19:37:48.633877 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.628539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.0.98.62:42099)
0924 19:37:48.628576 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.628578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.628697 (+ 119us) server_negotiation.cc:407] Connection header received
0924 19:37:48.628748 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.628749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.628756 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.628789 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:37:48.629291 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.629826 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.630513 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.630683 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.630732 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.631768 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:37:48.631769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:37:48.631770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:37:48.631775 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:37:48.633027 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.633327 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.633328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.633328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.633374 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:37:48.633507 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:37:48.633507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:37:48.633507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:37:48.633623 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:37:48.633690 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.633770 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.633820 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:37:48.635178 10669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:37:48.636938 10669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:37:48.642726 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.638818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.0.1:47854)
0924 19:37:48.638976 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.639013 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.639072 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.639256 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.639259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.639260 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:48.639320 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.639322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640756 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.640757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.641816 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.641817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642449 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.642525 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642539 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.642601 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.642648 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":29,"thread_start_us":61,"threads_started":1}
I20260924 19:37:48.642782 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.639013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46819 (local address 127.0.0.1:48518)
0924 19:37:48.639352 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.639360 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.639410 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.639597 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.639598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.639599 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:48.639645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.639646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640526 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.640527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.641766 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.641768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642558 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.642612 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642615 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.642659 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.642690 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":63,"threads_started":1}
I20260924 19:37:48.642781 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.639178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.0.98.3:46819)
0924 19:37:48.639327 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.639331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.639404 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:37:48.639514 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.639516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.639529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.639591 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:48.639693 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640443 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.641829 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642495 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642561 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642603 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.642669 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.642703 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260924 19:37:48.643102 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.638935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.0.98.1:34711)
0924 19:37:48.639129 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.639134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.639395 (+ 261us) server_negotiation.cc:407] Connection header received
0924 19:37:48.639511 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.639513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.639528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.639616 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:48.639812 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640848 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.641801 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642496 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642539 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642602 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.642987 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.643028 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260924 19:37:48.643277 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.638826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34711 (local address 127.0.0.1:54130)
0924 19:37:48.639323 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.639334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.639421 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.639693 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.639694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.639696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:48.639735 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.639736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640929 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.640930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.641719 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.641721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642659 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.642716 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642720 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.643170 (+ 450us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.643214 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":12,"thread_start_us":52,"threads_started":1}
I20260924 19:37:48.643342 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.638925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.98.2:35573)
0924 19:37:48.639103 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.639108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.639118 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:37:48.639155 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.639157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.639172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.639258 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:48.639384 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.640670 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.641909 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.642371 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.642430 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.642487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.643228 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.643270 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260924 19:37:48.643455 11062 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5d790dbcac4615b85c15a92078e265 (DEFAULT_TABLE table=test-table [id=1ca9f7f6e6be41658a05de90b09513ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:48.643785 11062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5d790dbcac4615b85c15a92078e265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:48.643977 10794 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5d790dbcac4615b85c15a92078e265 (DEFAULT_TABLE table=test-table [id=1ca9f7f6e6be41658a05de90b09513ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:48.644272 10794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5d790dbcac4615b85c15a92078e265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:48.644383 10928 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5d790dbcac4615b85c15a92078e265 (DEFAULT_TABLE table=test-table [id=1ca9f7f6e6be41658a05de90b09513ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:37:48.644690 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5d790dbcac4615b85c15a92078e265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:37:48.647241 11142 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: Bootstrap starting.
I20260924 19:37:48.648062 11142 tablet_bootstrap.cc:654] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:48.648675 11142 log.cc:824] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:48.649731 11142 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: No bootstrap required, opened a new log
I20260924 19:37:48.649798 11142 ts_tablet_manager.cc:1397] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 19:37:48.651227 11143 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Bootstrap starting.
I20260924 19:37:48.651685 11142 raft_consensus.cc:348] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.651889 11142 raft_consensus.cc:374] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:48.651904 11143 tablet_bootstrap.cc:654] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:48.651960 11142 raft_consensus.cc:729] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7a3d50875d41c1972bc124b66b7a03, State: Initialized, Role: FOLLOWER
I20260924 19:37:48.652108 11142 consensus_queue.cc:260] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.652405 11142 ts_tablet_manager.cc:1428] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:37:48.652450 11143 log.cc:824] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:48.652544 10860 heartbeater.cc:503] Master 127.0.98.62:42099 was elected leader, sending a full tablet report...
I20260924 19:37:48.652561 11145 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Bootstrap starting.
I20260924 19:37:48.653110 11143 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: No bootstrap required, opened a new log
I20260924 19:37:48.653177 11143 ts_tablet_manager.cc:1397] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:48.653213 11145 tablet_bootstrap.cc:654] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Neither blocks nor log segments found. Creating new log.
I20260924 19:37:48.653666 11145 log.cc:824] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Log is configured to *not* fsync() on all Append() calls
I20260924 19:37:48.654276 11145 tablet_bootstrap.cc:492] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: No bootstrap required, opened a new log
I20260924 19:37:48.654332 11145 ts_tablet_manager.cc:1397] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:37:48.654633 11143 raft_consensus.cc:348] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.654763 11143 raft_consensus.cc:374] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:48.654784 11143 raft_consensus.cc:729] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b83ea9c696a494a8f8e141ef27b7d3a, State: Initialized, Role: FOLLOWER
I20260924 19:37:48.654865 11143 consensus_queue.cc:260] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.655068 11143 ts_tablet_manager.cc:1428] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:37:48.655114 10994 heartbeater.cc:503] Master 127.0.98.62:42099 was elected leader, sending a full tablet report...
I20260924 19:37:48.656045 11145 raft_consensus.cc:348] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.656178 11145 raft_consensus.cc:374] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:37:48.656207 11145 raft_consensus.cc:729] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54945acd51644d6993b9dcfd98f57f24, State: Initialized, Role: FOLLOWER
I20260924 19:37:48.656327 11145 consensus_queue.cc:260] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.656585 11145 ts_tablet_manager.cc:1428] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:37:48.656625 11128 heartbeater.cc:503] Master 127.0.98.62:42099 was elected leader, sending a full tablet report...
W20260924 19:37:48.723961 10995 tablet.cc:2369] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:37:48.770889 11150 raft_consensus.cc:482] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:37:48.771054 11150 raft_consensus.cc:504] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.771399 11150 leader_election.cc:288] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b83ea9c696a494a8f8e141ef27b7d3a (127.0.98.2:35573), 0b7a3d50875d41c1972bc124b66b7a03 (127.0.98.1:34711)
I20260924 19:37:48.774065 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.771649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51585 (local address 127.0.98.2:35573)
0924 19:37:48.771706 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.771708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.771741 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:37:48.771770 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.771771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.771784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.771832 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:48.772001 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.772470 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773393 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.773755 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773813 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.773824 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.773963 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.773996 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:37:48.774065 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.771571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.98.3:51585)
0924 19:37:48.771621 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.771633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.771705 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.771848 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.771850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.771851 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:48.771913 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.771915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.772543 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.772545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.773311 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.773313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.773825 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.773903 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.773906 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.773949 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.773989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":41}
I20260924 19:37:48.774135 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.771725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47853 (local address 127.0.98.1:34711)
0924 19:37:48.771765 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.771767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.771851 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:37:48.771896 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.771897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.771907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.771990 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:37:48.772133 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.772591 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773479 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.773830 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773886 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.773897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.774039 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.774074 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:37:48.774340 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.771574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34711 (local address 127.0.98.3:47853)
0924 19:37:48.771799 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.771807 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:37:48.771846 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:37:48.772006 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:37:48.772006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:37:48.772008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:37:48.772074 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.772075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.772660 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.772661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.773396 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:37:48.773398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.773904 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.773905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:37:48.773977 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.773980 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:37:48.774022 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:37:48.774058 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":76,"thread_start_us":96,"threads_started":1}
I20260924 19:37:48.774477 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5d790dbcac4615b85c15a92078e265" candidate_uuid: "54945acd51644d6993b9dcfd98f57f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" is_pre_election: true
I20260924 19:37:48.774641 10948 raft_consensus.cc:2454] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54945acd51644d6993b9dcfd98f57f24 in term 0.
I20260924 19:37:48.774632 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5d790dbcac4615b85c15a92078e265" candidate_uuid: "54945acd51644d6993b9dcfd98f57f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7a3d50875d41c1972bc124b66b7a03" is_pre_election: true
I20260924 19:37:48.774760 10814 raft_consensus.cc:2454] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54945acd51644d6993b9dcfd98f57f24 in term 0.
I20260924 19:37:48.774880 11017 leader_election.cc:302] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54945acd51644d6993b9dcfd98f57f24, 8b83ea9c696a494a8f8e141ef27b7d3a; no voters:
I20260924 19:37:48.775023 11150 raft_consensus.cc:2781] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:37:48.775085 11150 raft_consensus.cc:482] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:37:48.775107 11150 raft_consensus.cc:3037] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:48.775720 11150 raft_consensus.cc:504] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.775826 11150 leader_election.cc:288] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [CANDIDATE]: Term 1 election: Requested vote from peers 8b83ea9c696a494a8f8e141ef27b7d3a (127.0.98.2:35573), 0b7a3d50875d41c1972bc124b66b7a03 (127.0.98.1:34711)
I20260924 19:37:48.776001 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5d790dbcac4615b85c15a92078e265" candidate_uuid: "54945acd51644d6993b9dcfd98f57f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7a3d50875d41c1972bc124b66b7a03"
I20260924 19:37:48.776037 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5d790dbcac4615b85c15a92078e265" candidate_uuid: "54945acd51644d6993b9dcfd98f57f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a"
I20260924 19:37:48.776094 10948 raft_consensus.cc:3037] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:48.776088 10814 raft_consensus.cc:3037] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:37:48.776909 10948 raft_consensus.cc:2454] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54945acd51644d6993b9dcfd98f57f24 in term 1.
I20260924 19:37:48.776909 10814 raft_consensus.cc:2454] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54945acd51644d6993b9dcfd98f57f24 in term 1.
I20260924 19:37:48.777112 11015 leader_election.cc:302] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7a3d50875d41c1972bc124b66b7a03, 54945acd51644d6993b9dcfd98f57f24; no voters:
I20260924 19:37:48.777184 11150 raft_consensus.cc:2781] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:37:48.777297 11150 raft_consensus.cc:686] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 1 LEADER]: Becoming Leader. State: Replica: 54945acd51644d6993b9dcfd98f57f24, State: Running, Role: LEADER
I20260924 19:37:48.777377 11150 consensus_queue.cc:237] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } }
I20260924 19:37:48.778190 10669 catalog_manager.cc:5495] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54945acd51644d6993b9dcfd98f57f24 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54945acd51644d6993b9dcfd98f57f24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54945acd51644d6993b9dcfd98f57f24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46819 } health_report { overall_health: HEALTHY } } }
I20260924 19:37:48.787550 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.785460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.3:46819)
0924 19:37:48.785509 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:37:48.785510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:37:48.785518 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:37:48.785544 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:37:48.785545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:37:48.785563 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:37:48.785620 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:37:48.785733 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.786175 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.786879 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:37:48.787064 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:37:48.787114 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:37:48.787265 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:37:48.787353 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:37:48.787420 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:37:48.787454 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:37:48.787472 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:37:48.789139 10948 raft_consensus.cc:1261] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 1 FOLLOWER]: Refusing update from remote peer 54945acd51644d6993b9dcfd98f57f24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:48.789148 10814 raft_consensus.cc:1261] T 6b5d790dbcac4615b85c15a92078e265 P 0b7a3d50875d41c1972bc124b66b7a03 [term 1 FOLLOWER]: Refusing update from remote peer 54945acd51644d6993b9dcfd98f57f24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:37:48.789490 11152 consensus_queue.cc:1035] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7a3d50875d41c1972bc124b66b7a03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:48.789608 11150 consensus_queue.cc:1035] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b83ea9c696a494a8f8e141ef27b7d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:37:48.794023 10668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:48.794111 10668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:37:48.794796 10668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22acaf1570c740b7b2fcce73205e49df: Sending DeleteTablet for 3 replicas of tablet 6b5d790dbcac4615b85c15a92078e265
I20260924 19:37:48.795167 10928 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5d790dbcac4615b85c15a92078e265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47854
I20260924 19:37:48.795159 10794 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5d790dbcac4615b85c15a92078e265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54130
I20260924 19:37:48.795369 11062 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5d790dbcac4615b85c15a92078e265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:37:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48518
I20260924 19:37:48.795503 11164 tablet_replica.cc:330] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: stopping tablet replica
I20260924 19:37:48.795588 11164 raft_consensus.cc:2229] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:37:48.795655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10731
I20260924 19:37:48.796551 11164 raft_consensus.cc:2258] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:48.796757 11166 tablet_replica.cc:330] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: stopping tablet replica
I20260924 19:37:48.796841 11166 raft_consensus.cc:2229] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:37:48.796947 11166 raft_consensus.cc:2258] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:37:48.796973 11164 ts_tablet_manager.cc:1905] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:48.797325 11166 ts_tablet_manager.cc:1905] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:37:48.798633 11164 ts_tablet_manager.cc:1918] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:48.798712 11164 log.cc:1196] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/6b5d790dbcac4615b85c15a92078e265
I20260924 19:37:48.798851 11166 ts_tablet_manager.cc:1918] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:37:48.798921 11166 log.cc:1196] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/6b5d790dbcac4615b85c15a92078e265
I20260924 19:37:48.799000 11164 ts_tablet_manager.cc:1939] T 6b5d790dbcac4615b85c15a92078e265 P 8b83ea9c696a494a8f8e141ef27b7d3a: Deleting consensus metadata
I20260924 19:37:48.799181 11166 ts_tablet_manager.cc:1939] T 6b5d790dbcac4615b85c15a92078e265 P 54945acd51644d6993b9dcfd98f57f24: Deleting consensus metadata
I20260924 19:37:48.799459 10655 catalog_manager.cc:4841] TS 8b83ea9c696a494a8f8e141ef27b7d3a (127.0.98.2:35573): tablet 6b5d790dbcac4615b85c15a92078e265 (table test-table [id=1ca9f7f6e6be41658a05de90b09513ea]) successfully deleted
I20260924 19:37:48.799784 10655 catalog_manager.cc:4841] TS 54945acd51644d6993b9dcfd98f57f24 (127.0.98.3:46819): tablet 6b5d790dbcac4615b85c15a92078e265 (table test-table [id=1ca9f7f6e6be41658a05de90b09513ea]) successfully deleted
W20260924 19:37:48.801898 10653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34711 (error 108)
I20260924 19:37:48.802341 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:37:48.802228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34711 (local address 127.0.0.1:54136)
0924 19:37:48.802273 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:37:48.802309 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:37:48.802397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10865
W20260924 19:37:48.802474 10653 catalog_manager.cc:4584] TS 0b7a3d50875d41c1972bc124b66b7a03 (127.0.98.1:34711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b5d790dbcac4615b85c15a92078e265: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34711: connect: Connection refused (error 111)
I20260924 19:37:48.806517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10999
I20260924 19:37:48.810272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 10637
2026-09-24T19:37:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1832 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[11170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[11170](info): set up 2 sockets
Sep 24 19:37:48 dist-test-slave-6kf3 krb5kdc[11170](info): commencing operation
krb5kdc: starting...
W20260924 19:37:49.752022 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:37:49 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:37:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 19:37:49.894995 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskm9q4Up/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/postgres -l logfile start
2026-09-24 19:37:52.083 UTC [11200] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:37:52.083 UTC [11200] LOG: listening on IPv4 address "127.0.98.20", port 49759
2026-09-24 19:37:52.086 UTC [11200] LOG: listening on Unix socket "/tmp/.s.PGSQL.49759"
2026-09-24 19:37:52.106 UTC [11203] LOG: database system was shut down at 2026-09-24 19:37:50 UTC
2026-09-24 19:37:52.109 UTC [11200] LOG: database system is ready to accept connections
I20260924 19:37:52.971674 392 mini_postgres.cc:96] Postgres bound to 49759
2026-09-24 19:37:52.977 UTC [11211] FATAL: database "slave" does not exist
127.0.98.20:49759 - accepting connections
I20260924 19:37:52.977734 392 mini_ranger.cc:161] Starting Ranger...
I20260924 19:37:52.990899 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:37:53.417275 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:37:53.417341 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin
2026-09-24 19:37:53,688 [I] DB FLAVOR :POSTGRES
2026-09-24 19:37:53,689 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:37:53,689 [I] Checking connection..
2026-09-24 19:37:53,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:37:53,928 [I] Checking connection passed.
2026-09-24 19:37:53,928 [I] --------- Verifying version history table ---------
2026-09-24 19:37:53,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:37:54,151 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:37:54,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:37:54,399 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:37:54,399 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:37:54,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:37:54.641 UTC [11275] WARNING: there is no transaction in progress
2026-09-24 19:37:54,646 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:37:54,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:37:54,875 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:37:54,875 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:37:54,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:37:55,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:37:55,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6kf3.c.gcp-upstream.internal','N') ;"
2026-09-24 19:37:55,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:37:55,838 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:37:55,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:37:56,077 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:37:56,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:37:56,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:37:56,564 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:37:56,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:37:57.434 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.449 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.464 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.479 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.493 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.517 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.578 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.585 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.598 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.607 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.616 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.629 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.638 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.644 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.650 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:57.656 UTC [11383] WARNING: there is no transaction in progress
2026-09-24 19:37:58,226 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:37:58,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:37:58,469 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:37:58,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:37:58,716 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:37:58,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:37:58,948 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:37:58,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:37:59,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:37:59,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6kf3.c.gcp-upstream.internal';"
2026-09-24 19:37:59,651 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:37:59,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:37:59,904 [I] DB_PATCHES have already been applied
I20260924 19:37:59.909325 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskm9q4Up/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:37:59.909387 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 19:37:59.912117 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56715
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:38:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:38:00.908555 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.931s user 0.003s sys 0.004s
Sep 24, 2026 7:38:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:38:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:38:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:38:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:38:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:38:20.473395 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:38:20 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278700, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:38:20.544540 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:38:20.552655 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:38:21.284238 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:38:21.284317 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms
I20260924 19:38:21.284332 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49759
2026-09-24 19:38:21,439 [I] DB FLAVOR :POSTGRES
2026-09-24 19:38:21,439 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:38:21,439 [I] Checking connection
2026-09-24 19:38:21,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:38:21,691 [I] connection success
2026-09-24 19:38:21,692 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:38:21,692 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:38:21,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:38:21,912 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:38:21,913 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:38:21,913 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:38:21,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:38:22,161 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:38:22.573149 392 mini_ranger_kms.cc:310] Created kms service
I20260924 19:38:22.796967 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:38:22.835572 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:38:22 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278700, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:23.195524 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:38:23.195631 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:38:23.195649 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 19:38:23.199136 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11563
I20260924 19:38:23.199230 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45637
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:38:25.215477 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.671s user 0.000s sys 0.137s
I20260924 19:38:25.215586 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:38:25.215673 392 mini_ranger_kms.cc:401] 127.0.98.20:45637/kms/v1/keys
Sep 24 19:38:25 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:38:25 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:38:26 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:38:28 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:38:28.480688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42327
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45637/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:38:28.559244 11614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:38:28.559434 11614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:38:28.559453 11614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:38:28.560531 11614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:38:28.560563 11614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:38:28.560576 11614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:38:28.560587 11614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:38:28.560597 11614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:38:28.562012 11614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45637/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42327
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:38:28.562191 11614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:38:28.562351 11614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:38:28.566998 11622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:28.567085 11619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:28.567157 11614 server_base.cc:1034] running on GCE node
W20260924 19:38:28.567164 11620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:28.567521 11614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:38:28.567796 11614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:38:28.568951 11614 hybrid_clock.cc:648] HybridClock initialized: now 1790278708568928 us; error 36 us; skew 500 ppm
Sep 24 19:38:28 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:38:28.571786 11614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:38:28.572242 11614 webserver.cc:460] Webserver started at http://127.0.98.62:44497/ using document root <none> and password file <none>
I20260924 19:38:28.572448 11614 fs_manager.cc:359] Metadata directory not provided
I20260924 19:38:28.572490 11614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:38:28.572623 11614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:38:28 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:28.624540 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "9b0da3c21bd24b1babb907e2b8d9350d"
format_stamp: "Formatted at 2026-09-24 19:38:28 on dist-test-slave-6kf3"
server_key: "2a958c01cd7ee7628bcacbbb653d4a82"
server_key_iv: "0afdba7c6bc48bffa39309d647c51d86"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:28.624919 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b0da3c21bd24b1babb907e2b8d9350d"
format_stamp: "Formatted at 2026-09-24 19:38:28 on dist-test-slave-6kf3"
server_key: "2a958c01cd7ee7628bcacbbb653d4a82"
server_key_iv: "0afdba7c6bc48bffa39309d647c51d86"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:28.626309 11614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:38:28.641225 11629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:28.641469 11614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:38:28.641577 11614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "9b0da3c21bd24b1babb907e2b8d9350d"
format_stamp: "Formatted at 2026-09-24 19:38:28 on dist-test-slave-6kf3"
server_key: "2a958c01cd7ee7628bcacbbb653d4a82"
server_key_iv: "0afdba7c6bc48bffa39309d647c51d86"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:28.641669 11614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:38:28.666685 11614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:38:28.668167 11614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:38:28.668273 11614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:38:28.672428 11614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42327
I20260924 19:38:28.672477 11681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42327 every 8 connection(s)
I20260924 19:38:28.672798 11614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:38:28.673384 11682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:38:28.675539 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: Bootstrap starting.
I20260924 19:38:28.676180 11682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: Neither blocks nor log segments found. Creating new log.
I20260924 19:38:28.676199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 11614
I20260924 19:38:28.676283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:38:28.676476 392 external_mini_cluster.cc:1428] Setting key 00bfa62be754cd48a1e0e1914f1760a8
I20260924 19:38:28.676468 11682 log.cc:824] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:38:28.677197 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: No bootstrap required, opened a new log
I20260924 19:38:28.679191 11682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } }
I20260924 19:38:28.679319 11682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:38:28.679349 11682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0da3c21bd24b1babb907e2b8d9350d, State: Initialized, Role: FOLLOWER
I20260924 19:38:28.679476 11682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } }
I20260924 19:38:28.679548 11682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:38:28.679577 11682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:38:28.679622 11682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:38:28.680610 11682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } }
I20260924 19:38:28.680712 11682 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0da3c21bd24b1babb907e2b8d9350d; no voters:
I20260924 19:38:28.680862 11682 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:38:28.680893 11687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:38:28.681020 11687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [term 1 LEADER]: Becoming Leader. State: Replica: 9b0da3c21bd24b1babb907e2b8d9350d, State: Running, Role: LEADER
I20260924 19:38:28.681113 11682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:38:28.681141 11687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } }
I20260924 19:38:28.681555 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0da3c21bd24b1babb907e2b8d9350d. Latest consensus state: current_term: 1 leader_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } } }
I20260924 19:38:28.681697 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [sys.catalog]: This master's current role is: LEADER
I20260924 19:38:28.682433 11689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0da3c21bd24b1babb907e2b8d9350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42327 } } }
I20260924 19:38:28.682510 11689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d [sys.catalog]: This master's current role is: LEADER
I20260924 19:38:28.682601 11682 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 19:38:28 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:38:28.684613 11696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:38:28.685576 11696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:38:28.687556 11696 catalog_manager.cc:1347] Generated new cluster ID: e4c17fba01844a88927215e40fbebb20
I20260924 19:38:28.687601 11696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:38:28.688994 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:28.676963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.0.98.62:42327)
0924 19:38:28.677132 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:28.677142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:28.677173 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:38:28.677618 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:28.677625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:28.677674 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:28.677793 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:38:28.678387 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:28.679237 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:28.680142 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:28.680570 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:28.680662 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:28.684274 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:38:28.684322 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:38:28.684323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:38:28.684412 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:38:28.686913 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:28.687304 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:28.687308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:28.687308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:28.687400 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:28.687606 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:28.687606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:28.687606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:28.687965 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:38:28.688560 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:28.688635 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:38:28.688780 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260924 19:38:28.706633 11696 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:38:28.707405 11696 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:38:28.709877 11696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: Generated new TSK 0
I20260924 19:38:28.710033 11696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:38:29.401959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:38:29.490589 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:38:29.490770 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:38:29.490798 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:38:29.491882 11727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:38:29.491920 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:38:29.491981 11727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:38:29.493418 11727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:38:29.493637 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:38:29.493830 11727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:38:29.498387 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:29.498391 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:29.498519 11727 server_base.cc:1034] running on GCE node
W20260924 19:38:29.498682 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:29.498883 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:38:29.499166 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:38:29.500381 11727 hybrid_clock.cc:648] HybridClock initialized: now 1790278709500354 us; error 41 us; skew 500 ppm
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:38:29.503259 11727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:38:29.503793 11727 webserver.cc:460] Webserver started at http://127.0.98.1:40339/ using document root <none> and password file <none>
I20260924 19:38:29.504093 11727 fs_manager.cc:359] Metadata directory not provided
I20260924 19:38:29.504163 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:38:29.504300 11727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:29.519254 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3896ca48ea94bd595903685d067a1ed"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "0867b4ad45a3bb947d7158ebb0b8d70d"
server_key_iv: "52d70289a4c10fcc9e314415b582bea9"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.519665 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3896ca48ea94bd595903685d067a1ed"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "0867b4ad45a3bb947d7158ebb0b8d70d"
server_key_iv: "52d70289a4c10fcc9e314415b582bea9"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.521283 11727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:38:29.536059 11744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.536265 11727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 19:38:29.536347 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "a3896ca48ea94bd595903685d067a1ed"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "0867b4ad45a3bb947d7158ebb0b8d70d"
server_key_iv: "52d70289a4c10fcc9e314415b582bea9"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.536437 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:38:29.550896 11727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:38:29.552682 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:38:29.552815 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:38:29.553059 11727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:38:29.553448 11727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:38:29.553488 11727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.553522 11727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:38:29.553565 11727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.561308 11727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33827
I20260924 19:38:29.561374 11857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33827 every 8 connection(s)
I20260924 19:38:29.561753 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:38:29.567516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 11727
I20260924 19:38:29.567634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:38:29.567847 392 external_mini_cluster.cc:1428] Setting key 224d9e876f8991be575b72c19a92fd27
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:38:29.572065 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:29.562534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50729 (local address 127.0.98.62:42327)
0924 19:38:29.562718 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:29.562721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:29.563243 (+ 522us) server_negotiation.cc:407] Connection header received
0924 19:38:29.563746 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:29.563747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:29.563762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:29.563839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:38:29.564453 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.564966 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.565982 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.566194 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.566256 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:29.569217 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:38:29.569219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:38:29.569219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:38:29.569228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:38:29.570782 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:29.571143 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:29.571144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:29.571144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:29.571197 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:29.571395 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:29.571396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:29.571397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:29.571540 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:38:29.571595 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:29.571856 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 19:38:29.571972 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260924 19:38:29.572230 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:29.562503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42327 (local address 127.0.98.1:50729)
0924 19:38:29.563097 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:29.563146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:29.563648 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:29.563921 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:29.563926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:29.564145 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:38:29.564366 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:29.564372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.565052 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.565053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:29.565897 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:29.565900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.566268 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.566269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:29.566402 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:29.566948 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:38:29.566956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:38:29.569109 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:38:29.570859 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:29.570865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:29.570867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:29.571078 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:29.571249 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:29.571249 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:29.571250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:29.571344 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:29.571598 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:38:29.571605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:38:29.571740 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:38:29.571826 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:38:29.572015 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":512,"threads_started":1}
I20260924 19:38:29.572748 11858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42327
I20260924 19:38:29.572834 11858 heartbeater.cc:464] Registering TS with master...
I20260924 19:38:29.573006 11858 heartbeater.cc:511] Master 127.0.98.62:42327 requested a full tablet report, sending...
I20260924 19:38:29.573464 11646 ts_manager.cc:194] Registered new tserver with Master: a3896ca48ea94bd595903685d067a1ed (127.0.98.1:33827)
I20260924 19:38:29.574062 11646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50729
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:38:29.594637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:38:29.707289 11864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:38:29.707620 11864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:38:29.707680 11864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:38:29.709302 11864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:38:29.709386 11864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:38:29.709442 11864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:38:29.711701 11864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:38:29.712074 11864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:38:29.712339 11864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:38:29.718230 11874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:29.718281 11872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:29.720094 11871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:29.720504 11864 server_base.cc:1034] running on GCE node
I20260924 19:38:29.720712 11864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:38:29.720973 11864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:38:29.724241 11864 hybrid_clock.cc:648] HybridClock initialized: now 1790278709724217 us; error 38 us; skew 500 ppm
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:38:29.727497 11864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:38:29.728040 11864 webserver.cc:460] Webserver started at http://127.0.98.2:44817/ using document root <none> and password file <none>
I20260924 19:38:29.728299 11864 fs_manager.cc:359] Metadata directory not provided
I20260924 19:38:29.728348 11864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:38:29.728471 11864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:29.742910 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "346ff890b2b84744a92153f162b549e8"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "d218a1fb99e6d665569421c130436e6f"
server_key_iv: "20596d7d33fc6fabd472a4a38385497b"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.743463 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "346ff890b2b84744a92153f162b549e8"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "d218a1fb99e6d665569421c130436e6f"
server_key_iv: "20596d7d33fc6fabd472a4a38385497b"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.745935 11864 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:38:29.758599 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.758977 11864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:38:29.759205 11864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "346ff890b2b84744a92153f162b549e8"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "d218a1fb99e6d665569421c130436e6f"
server_key_iv: "20596d7d33fc6fabd472a4a38385497b"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.759385 11864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:38:29.779634 11864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:38:29.781667 11864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:38:29.781793 11864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:38:29.782042 11864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:38:29.782397 11864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:38:29.782431 11864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.782459 11864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:38:29.782477 11864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:29.793725 11864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36883
I20260924 19:38:29.794241 11864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:38:29.797039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 11864
I20260924 19:38:29.797134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:38:29.797333 392 external_mini_cluster.cc:1428] Setting key f8328bd1b3ccfc4f7cbe0beb1a694445
I20260924 19:38:29.799834 11995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36883 every 8 connection(s)
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:38:29.821528 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:29.797351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54019 (local address 127.0.98.62:42327)
0924 19:38:29.797442 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:29.797444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:29.800472 (+ 3028us) server_negotiation.cc:407] Connection header received
0924 19:38:29.803414 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:29.803415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:29.803427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:29.803455 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:38:29.804407 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.805194 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.810464 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.810797 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.810846 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:29.818388 (+ 7542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:38:29.818390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:38:29.818390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:38:29.818399 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:38:29.819979 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:29.820452 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:29.820454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:29.820454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:29.820510 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:29.820761 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:29.820761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:29.820762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:29.820893 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:38:29.820942 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:29.821339 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 19:38:29.821427 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:38:29.822288 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:29.797033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42327 (local address 127.0.98.2:54019)
0924 19:38:29.800340 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:29.800385 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:29.803289 (+ 2904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:29.803637 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:29.803641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:29.803872 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:38:29.804306 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:29.804313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.805302 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.805302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:29.806545 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:29.806548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:29.810943 (+ 4395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:29.810944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:29.811111 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:29.812050 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:38:29.812071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:38:29.818264 (+ 6193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:38:29.820076 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:29.820084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:29.820087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:29.820366 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:29.820569 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:29.820570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:29.820570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:29.820696 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:29.820972 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:38:29.820979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:38:29.821179 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 19:38:29.821871 (+ 692us) client_negotiation.cc:240] Negotiation successful
0924 19:38:29.822041 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3263,"thread_start_us":89,"threads_started":1}
I20260924 19:38:29.822917 11996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42327
I20260924 19:38:29.823005 11996 heartbeater.cc:464] Registering TS with master...
I20260924 19:38:29.823194 11996 heartbeater.cc:511] Master 127.0.98.62:42327 requested a full tablet report, sending...
I20260924 19:38:29.823539 11646 ts_manager.cc:194] Registered new tserver with Master: 346ff890b2b84744a92153f162b549e8 (127.0.98.2:36883)
I20260924 19:38:29.824049 11646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54019
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:38:29.844576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:38:29.959111 12001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:38:29.959446 12001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:38:29.959528 12001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:38:29.961244 12001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:38:29.961347 12001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:38:29.961426 12001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:38:29.963595 12001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:38:29.964082 12001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:38:29.964375 12001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:38:29.971370 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:29.971321 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:38:29.972188 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:38:29.972321 12001 server_base.cc:1034] running on GCE node
I20260924 19:38:29.972564 12001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:38:29.972882 12001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:38:29.974206 12001 hybrid_clock.cc:648] HybridClock initialized: now 1790278709974185 us; error 34 us; skew 500 ppm
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:38:29.977258 12001 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:38:29.977757 12001 webserver.cc:460] Webserver started at http://127.0.98.3:45915/ using document root <none> and password file <none>
I20260924 19:38:29.978021 12001 fs_manager.cc:359] Metadata directory not provided
I20260924 19:38:29.978077 12001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:38:29.978216 12001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:38:29 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:29.992588 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "0973b4489a414189a41fab48f6eb525b"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "181c6c6891a2921b70912daa7d0ce391"
server_key_iv: "bd3ad77e3e658ac163b8e1bb7d3c713e"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.993063 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0973b4489a414189a41fab48f6eb525b"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "181c6c6891a2921b70912daa7d0ce391"
server_key_iv: "bd3ad77e3e658ac163b8e1bb7d3c713e"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:29.994724 12001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:38:30.018042 12016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:30.018306 12001 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.000s
I20260924 19:38:30.018420 12001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "0973b4489a414189a41fab48f6eb525b"
format_stamp: "Formatted at 2026-09-24 19:38:29 on dist-test-slave-6kf3"
server_key: "181c6c6891a2921b70912daa7d0ce391"
server_key_iv: "bd3ad77e3e658ac163b8e1bb7d3c713e"
server_key_version: "kuduclusterkey@0"
I20260924 19:38:30.018525 12001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:38:30.036966 12001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:38:30.039790 12001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:38:30.040031 12001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:38:30.040504 12001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:38:30.040944 12001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:38:30.041023 12001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:30.041087 12001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:38:30.041141 12001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:38:30.047122 12001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36483
I20260924 19:38:30.047595 12001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:38:30.050310 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36483 every 8 connection(s)
I20260924 19:38:30.052903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 12001
I20260924 19:38:30.053012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:38:30.053223 392 external_mini_cluster.cc:1428] Setting key 32364642bb88b8315abb07805726c9bb
Sep 24 19:38:30 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:38:30.062220 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.051304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37513 (local address 127.0.98.62:42327)
0924 19:38:30.051346 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.051348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.051841 (+ 493us) server_negotiation.cc:407] Connection header received
0924 19:38:30.052537 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.052538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.052551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.052578 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:38:30.054174 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.054923 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.056343 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.056622 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.056660 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.059224 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:38:30.059225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:38:30.059226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:38:30.059233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:38:30.060735 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:30.061205 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:30.061207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:30.061207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:30.061263 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:30.061503 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:30.061504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:30.061504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:30.061627 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:38:30.061666 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.062030 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.062114 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:38:30.063035 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.050863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42327 (local address 127.0.98.3:37513)
0924 19:38:30.051596 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.051752 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.052415 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.052755 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.052759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.053119 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:38:30.054083 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.054091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.055029 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.055030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.056253 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.056255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.056740 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.056740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.056879 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.057574 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:38:30.057584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:38:30.059129 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:38:30.060818 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:30.060825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:30.060827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:30.061123 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:30.061324 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:38:30.061325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:38:30.061325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:38:30.061441 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:38:30.061706 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:38:30.061711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:38:30.061876 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.062658 (+ 782us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.062813 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":228,"threads_started":1}
I20260924 19:38:30.063649 12130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42327
I20260924 19:38:30.063735 12130 heartbeater.cc:464] Registering TS with master...
I20260924 19:38:30.064543 12130 heartbeater.cc:511] Master 127.0.98.62:42327 requested a full tablet report, sending...
I20260924 19:38:30.064936 11646 ts_manager.cc:194] Registered new tserver with Master: 0973b4489a414189a41fab48f6eb525b (127.0.98.3:36483)
I20260924 19:38:30.065415 11646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37513
I20260924 19:38:30.075173 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:38:30.088377 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.077740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.0.98.62:42327)
0924 19:38:30.077782 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.077784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.078051 (+ 267us) server_negotiation.cc:407] Connection header received
0924 19:38:30.078160 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.078161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.078171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.078198 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:38:30.079143 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.079890 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.081011 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.081281 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.081320 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.082844 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:38:30.082845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:38:30.082846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:38:30.082853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:38:30.087024 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:30.087583 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:30.087584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:30.087585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:30.087641 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:38:30.087813 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:38:30.087814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:38:30.087814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:38:30.087968 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:38:30.088038 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.088221 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.088292 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:38:30.090871 11646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:38:30.093214 11646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:38:30.106148 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.096849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.0.1:41906)
0924 19:38:30.098065 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.098073 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.098110 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.098429 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.098432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.098434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:38:30.098499 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.098500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.100017 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.100018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.101259 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.101262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.103755 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.103756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.103843 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.103860 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.104929 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.104991 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":72,"threads_started":1}
I20260924 19:38:30.106787 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.097099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.98.2:36883)
0924 19:38:30.097773 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.097778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.098187 (+ 409us) server_negotiation.cc:407] Connection header received
0924 19:38:30.098240 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.098243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.098260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.098331 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:38:30.098565 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.099889 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.101364 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.103192 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.103258 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.106625 (+ 3367us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.106661 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.106708 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":70,"threads_started":1}
I20260924 19:38:30.108242 11920 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba430a0fc3546ada5514b12300e94ff (DEFAULT_TABLE table=test-table [id=fd1bfa99bc93414aae51da52eece1079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:38:30.108610 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba430a0fc3546ada5514b12300e94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:38:30.109509 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.097335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36483 (local address 127.0.0.1:35758)
0924 19:38:30.104000 (+ 6665us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.104012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.104074 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.104443 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.104444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.104447 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:38:30.104501 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.104502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.105974 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.105975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.107050 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.107052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.109268 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.109269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.109343 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.109348 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.109394 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.109432 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6645,"thread_start_us":55,"threads_started":1}
I20260924 19:38:30.109714 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.097563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.0.98.3:36483)
0924 19:38:30.104198 (+ 6635us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.104203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.104212 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:38:30.104249 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.104251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.104266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.104332 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:38:30.104828 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.105868 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.108329 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.109088 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.109159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.109234 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.109600 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.109641 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6595,"thread_start_us":88,"threads_started":1}
I20260924 19:38:30.112726 12064 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba430a0fc3546ada5514b12300e94ff (DEFAULT_TABLE table=test-table [id=fd1bfa99bc93414aae51da52eece1079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:38:30.113041 12064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba430a0fc3546ada5514b12300e94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:38:30.118520 12149 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Bootstrap starting.
I20260924 19:38:30.119474 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.096628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33827 (local address 127.0.0.1:38384)
0924 19:38:30.097837 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.097892 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.097966 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.104779 (+ 6813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.104780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.104783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:38:30.104839 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.104840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.117026 (+ 12186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.117027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.118201 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.118205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.119193 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.119194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.119273 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.119277 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.119344 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.119389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"thread_start_us":77,"threads_started":1}
I20260924 19:38:30.119489 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.096718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.0.98.1:33827)
0924 19:38:30.104619 (+ 7901us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.104625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.104634 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:38:30.104664 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.104666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.104680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.104753 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:38:30.107150 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.108220 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.118290 (+ 10070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.119089 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.119191 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.119258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.119361 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.119402 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7865,"thread_start_us":67,"threads_started":1}
I20260924 19:38:30.120503 12150 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Bootstrap starting.
I20260924 19:38:30.120435 11792 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba430a0fc3546ada5514b12300e94ff (DEFAULT_TABLE table=test-table [id=fd1bfa99bc93414aae51da52eece1079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:38:30.120798 11792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba430a0fc3546ada5514b12300e94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:38:30.122001 12149 tablet_bootstrap.cc:654] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Neither blocks nor log segments found. Creating new log.
I20260924 19:38:30.122339 12149 log.cc:824] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:38:30.121747 12150 tablet_bootstrap.cc:654] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Neither blocks nor log segments found. Creating new log.
I20260924 19:38:30.123008 12150 log.cc:824] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:38:30.123801 12149 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: No bootstrap required, opened a new log
I20260924 19:38:30.123908 12149 ts_tablet_manager.cc:1397] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:38:30.126223 12149 raft_consensus.cc:348] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.126521 12149 raft_consensus.cc:374] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:38:30.126597 12149 raft_consensus.cc:729] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346ff890b2b84744a92153f162b549e8, State: Initialized, Role: FOLLOWER
I20260924 19:38:30.126781 12149 consensus_queue.cc:260] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.127099 12149 ts_tablet_manager.cc:1428] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:38:30.127238 11996 heartbeater.cc:503] Master 127.0.98.62:42327 was elected leader, sending a full tablet report...
I20260924 19:38:30.128494 12153 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: Bootstrap starting.
I20260924 19:38:30.128691 12150 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: No bootstrap required, opened a new log
I20260924 19:38:30.128750 12150 ts_tablet_manager.cc:1397] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:38:30.129144 12153 tablet_bootstrap.cc:654] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: Neither blocks nor log segments found. Creating new log.
I20260924 19:38:30.129431 12153 log.cc:824] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: Log is configured to *not* fsync() on all Append() calls
I20260924 19:38:30.130767 12150 raft_consensus.cc:348] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.130975 12150 raft_consensus.cc:374] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:38:30.131047 12150 raft_consensus.cc:729] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0973b4489a414189a41fab48f6eb525b, State: Initialized, Role: FOLLOWER
I20260924 19:38:30.131225 12150 consensus_queue.cc:260] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.131531 12150 ts_tablet_manager.cc:1428] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:38:30.131980 12130 heartbeater.cc:503] Master 127.0.98.62:42327 was elected leader, sending a full tablet report...
I20260924 19:38:30.136611 12153 tablet_bootstrap.cc:492] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: No bootstrap required, opened a new log
I20260924 19:38:30.136759 12153 ts_tablet_manager.cc:1397] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:38:30.139272 12153 raft_consensus.cc:348] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.139842 12153 raft_consensus.cc:374] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:38:30.139958 12153 raft_consensus.cc:729] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3896ca48ea94bd595903685d067a1ed, State: Initialized, Role: FOLLOWER
I20260924 19:38:30.140158 12153 consensus_queue.cc:260] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.140625 12153 ts_tablet_manager.cc:1428] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 19:38:30.140836 11858 heartbeater.cc:503] Master 127.0.98.62:42327 was elected leader, sending a full tablet report...
I20260924 19:38:30.214572 12157 raft_consensus.cc:482] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:38:30.214715 12157 raft_consensus.cc:504] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.215085 12157 leader_election.cc:288] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0973b4489a414189a41fab48f6eb525b (127.0.98.3:36483), 346ff890b2b84744a92153f162b549e8 (127.0.98.2:36883)
I20260924 19:38:30.223455 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.215455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56085 (local address 127.0.98.2:36883)
0924 19:38:30.215498 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.215500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.217776 (+ 2276us) server_negotiation.cc:407] Connection header received
0924 19:38:30.217877 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.217878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.217890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.217917 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:38:30.218168 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.218873 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.222260 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.222837 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.222881 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.222893 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.223323 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.223368 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:38:30.224035 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.215262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.98.1:56085)
0924 19:38:30.217690 (+ 2428us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.217703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.217806 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.217983 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.217984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.217986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:38:30.218067 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.218069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.218991 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.218992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.220293 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.220296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.222975 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.222976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.223046 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.223049 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.223879 (+ 830us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.223947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2401,"thread_start_us":92,"threads_started":1}
Sep 24 19:38:30 dist-test-slave-6kf3 krb5kdc[11170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:38:30.225378 11950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba430a0fc3546ada5514b12300e94ff" candidate_uuid: "a3896ca48ea94bd595903685d067a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346ff890b2b84744a92153f162b549e8" is_pre_election: true
I20260924 19:38:30.225572 11950 raft_consensus.cc:2454] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3896ca48ea94bd595903685d067a1ed in term 0.
I20260924 19:38:30.225840 11747 leader_election.cc:302] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346ff890b2b84744a92153f162b549e8, a3896ca48ea94bd595903685d067a1ed; no voters:
I20260924 19:38:30.225970 12157 raft_consensus.cc:2781] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:38:30.226006 12157 raft_consensus.cc:482] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:38:30.226025 12157 raft_consensus.cc:3037] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:38:30.226909 12157 raft_consensus.cc:504] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.227042 12157 leader_election.cc:288] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [CANDIDATE]: Term 1 election: Requested vote from peers 0973b4489a414189a41fab48f6eb525b (127.0.98.3:36483), 346ff890b2b84744a92153f162b549e8 (127.0.98.2:36883)
I20260924 19:38:30.227437 11950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba430a0fc3546ada5514b12300e94ff" candidate_uuid: "a3896ca48ea94bd595903685d067a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346ff890b2b84744a92153f162b549e8"
I20260924 19:38:30.227517 11950 raft_consensus.cc:3037] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:38:30.228353 11950 raft_consensus.cc:2454] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3896ca48ea94bd595903685d067a1ed in term 1.
I20260924 19:38:30.228614 11747 leader_election.cc:302] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346ff890b2b84744a92153f162b549e8, a3896ca48ea94bd595903685d067a1ed; no voters:
I20260924 19:38:30.228727 12157 raft_consensus.cc:2781] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:38:30.228868 12157 raft_consensus.cc:686] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [term 1 LEADER]: Becoming Leader. State: Replica: a3896ca48ea94bd595903685d067a1ed, State: Running, Role: LEADER
I20260924 19:38:30.228952 12157 consensus_queue.cc:237] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260924 19:38:30.230655 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.215561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35089 (local address 127.0.98.3:36483)
0924 19:38:30.215619 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.215621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.220472 (+ 4851us) server_negotiation.cc:407] Connection header received
0924 19:38:30.220560 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.220561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.220572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.220597 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:38:30.220776 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.221466 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.229672 (+ 8206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.230260 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.230309 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.230321 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.230543 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.230588 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:38:30.231005 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.215289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36483 (local address 127.0.98.1:35089)
0924 19:38:30.220403 (+ 5114us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.220413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:38:30.220497 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:38:30.220648 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:38:30.220649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:38:30.220650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:38:30.220716 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.220717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.221574 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.221575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.225035 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:38:30.225038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.230378 (+ 5340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.230379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:38:30.230467 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.230471 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:38:30.230872 (+ 401us) client_negotiation.cc:240] Negotiation successful
0924 19:38:30.230917 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5096,"thread_start_us":83,"threads_started":1}
I20260924 19:38:30.231215 11646 catalog_manager.cc:5495] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed reported cstate change: term changed from 0 to 1, leader changed from <none> to a3896ca48ea94bd595903685d067a1ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a3896ca48ea94bd595903685d067a1ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3896ca48ea94bd595903685d067a1ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } health_report { overall_health: UNKNOWN } } }
I20260924 19:38:30.231451 12084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba430a0fc3546ada5514b12300e94ff" candidate_uuid: "a3896ca48ea94bd595903685d067a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0973b4489a414189a41fab48f6eb525b" is_pre_election: true
I20260924 19:38:30.231590 12084 raft_consensus.cc:2454] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3896ca48ea94bd595903685d067a1ed in term 0.
I20260924 19:38:30.231681 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba430a0fc3546ada5514b12300e94ff" candidate_uuid: "a3896ca48ea94bd595903685d067a1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0973b4489a414189a41fab48f6eb525b"
I20260924 19:38:30.231721 12083 raft_consensus.cc:3037] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:38:30.232499 12083 raft_consensus.cc:2454] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3896ca48ea94bd595903685d067a1ed in term 1.
I20260924 19:38:30.252839 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.249765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.98.1:33827)
0924 19:38:30.249808 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:38:30.249811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:38:30.249820 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:38:30.249867 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:38:30.249868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:38:30.249891 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:38:30.249972 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:38:30.250088 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.250790 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.251774 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:38:30.252049 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:38:30.252104 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:38:30.252280 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:38:30.252420 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:38:30.252490 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:38:30.252716 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:38:30.252736 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:38:30.255040 11950 raft_consensus.cc:1261] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 1 FOLLOWER]: Refusing update from remote peer a3896ca48ea94bd595903685d067a1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:38:30.255467 12157 consensus_queue.cc:1035] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "346ff890b2b84744a92153f162b549e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:38:30.256870 12083 raft_consensus.cc:1261] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 1 FOLLOWER]: Refusing update from remote peer a3896ca48ea94bd595903685d067a1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:38:30.257162 12160 consensus_queue.cc:1035] T 0ba430a0fc3546ada5514b12300e94ff P a3896ca48ea94bd595903685d067a1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0973b4489a414189a41fab48f6eb525b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:38:30.258584 12163 mvcc.cc:205] Tried to move back new op lower bound from 7332981597201678336 to 7332981597098336256. Current Snapshot: MvccSnapshot[applied={T|T < 7332981597201678336}]
I20260924 19:38:30.260695 12164 mvcc.cc:205] Tried to move back new op lower bound from 7332981597201678336 to 7332981597098336256. Current Snapshot: MvccSnapshot[applied={T|T < 7332981597201678336}]
I20260924 19:38:30.263073 11646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:38:30.263160 11646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:38:30.264073 11646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b0da3c21bd24b1babb907e2b8d9350d: Sending DeleteTablet for 3 replicas of tablet 0ba430a0fc3546ada5514b12300e94ff
I20260924 19:38:30.264443 11792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba430a0fc3546ada5514b12300e94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:38:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38384
I20260924 19:38:30.264691 11920 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba430a0fc3546ada5514b12300e94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:38:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41906
I20260924 19:38:30.265035 12064 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba430a0fc3546ada5514b12300e94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:38:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35758
I20260924 19:38:30.265641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 11727
I20260924 19:38:30.270532 12173 tablet_replica.cc:330] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: stopping tablet replica
I20260924 19:38:30.270674 12174 tablet_replica.cc:330] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: stopping tablet replica
I20260924 19:38:30.270860 12174 raft_consensus.cc:2229] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:38:30.271008 12173 raft_consensus.cc:2229] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:38:30.271170 12173 raft_consensus.cc:2258] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:38:30.271286 12174 pending_rounds.cc:62] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Trying to abort 2 pending ops.
I20260924 19:38:30.271359 12174 pending_rounds.cc:68] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332981597201678336 op_type: WRITE_OP write_request { tablet_id: "0ba430a0fc3546ada5514b12300e94ff" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\342\372\325\325\006\"8\n\ntest-admin\022*\n fd1bfa99bc93414aae51da52eece1079\020\001\030\001 \001(\001" signature: "<redacted>"";i\346:J\341\202i\317\324\362\'L\251\371\2664T\353\032\234\336\377\3604\277D\252oN\254\034\301\260\313\316\301$\347\331\r\256\3253D0x\350\007\177\320|\267\347W\026\0350\0313\370\036\225\374" signing_key_seq_num: 0 } } request_id { client_id: "6ffd8cd8b3324e198a4d86301fe4b139" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 19:38:30.275516 12173 ts_tablet_manager.cc:1905] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:38:30.275785 12174 pending_rounds.cc:68] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332981597098336256 op_type: NO_OP noop_request { }
I20260924 19:38:30.275964 12174 raft_consensus.cc:2866] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:38:30.276042 12174 raft_consensus.cc:2258] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:38:30.274916 12018 connection.cc:535] server connection from 127.0.98.1:35089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:38:30.278033 12174 ts_tablet_manager.cc:1905] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:38:30.279719 12174 ts_tablet_manager.cc:1918] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:38:30.280436 12173 ts_tablet_manager.cc:1918] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:38:30.280562 12173 log.cc:1196] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/0ba430a0fc3546ada5514b12300e94ff
I20260924 19:38:30.280871 12173 ts_tablet_manager.cc:1939] T 0ba430a0fc3546ada5514b12300e94ff P 346ff890b2b84744a92153f162b549e8: Deleting consensus metadata
I20260924 19:38:30.280997 12174 log.cc:1196] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/0ba430a0fc3546ada5514b12300e94ff
I20260924 19:38:30.281620 12174 ts_tablet_manager.cc:1939] T 0ba430a0fc3546ada5514b12300e94ff P 0973b4489a414189a41fab48f6eb525b: Deleting consensus metadata
I20260924 19:38:30.282161 11632 catalog_manager.cc:4841] TS 346ff890b2b84744a92153f162b549e8 (127.0.98.2:36883): tablet 0ba430a0fc3546ada5514b12300e94ff (table test-table [id=fd1bfa99bc93414aae51da52eece1079]) successfully deleted
I20260924 19:38:30.282761 11631 catalog_manager.cc:4841] TS 0973b4489a414189a41fab48f6eb525b (127.0.98.3:36483): tablet 0ba430a0fc3546ada5514b12300e94ff (table test-table [id=fd1bfa99bc93414aae51da52eece1079]) successfully deleted
W20260924 19:38:30.283106 11630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33827 (error 108)
I20260924 19:38:30.283759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 11864
I20260924 19:38:30.287012 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:38:30.286891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33827 (local address 127.0.0.1:38398)
0924 19:38:30.286934 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:38:30.286978 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:38:30.287132 11630 catalog_manager.cc:4584] TS a3896ca48ea94bd595903685d067a1ed (127.0.98.1:33827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ba430a0fc3546ada5514b12300e94ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33827: connect: Connection refused (error 111)
I20260924 19:38:30.289242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 12001
I20260924 19:38:30.295190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 11614
I20260924 19:38:30.302404 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:38:30.636168 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:38:30.636266 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:38:30.970147 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:38:30.970 UTC [11200] LOG: received smart shutdown request
2026-09-24 19:38:30.973 UTC [11200] LOG: background worker "logical replication launcher" (PID 11209) exited with exit code 1
2026-09-24 19:38:30.973 UTC [11204] LOG: shutting down
2026-09-24 19:38:31.140 UTC [11200] LOG: database system is shut down
2026-09-24T19:38:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42408 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:38:31 dist-test-slave-6kf3 krb5kdc[12184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:38:31 dist-test-slave-6kf3 krb5kdc[12184](info): set up 2 sockets
Sep 24 19:38:31 dist-test-slave-6kf3 krb5kdc[12184](info): commencing operation
krb5kdc: starting...
W20260924 19:38:32.175539 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:38:32 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:38:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 19:38:32.320555 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskm9q4Up/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/postgres -l logfile start
2026-09-24 19:38:34.497 UTC [12214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:38:34.497 UTC [12214] LOG: listening on IPv4 address "127.0.98.20", port 57213
2026-09-24 19:38:34.501 UTC [12214] LOG: listening on Unix socket "/tmp/.s.PGSQL.57213"
2026-09-24 19:38:34.519 UTC [12217] LOG: database system was shut down at 2026-09-24 19:38:33 UTC
2026-09-24 19:38:34.522 UTC [12214] LOG: database system is ready to accept connections
I20260924 19:38:35.399595 392 mini_postgres.cc:96] Postgres bound to 57213
2026-09-24 19:38:35.405 UTC [12225] FATAL: database "slave" does not exist
127.0.98.20:57213 - accepting connections
I20260924 19:38:35.405731 392 mini_ranger.cc:161] Starting Ranger...
I20260924 19:38:35.420537 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:38:35.839496 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:38:35.839568 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin
2026-09-24 19:38:36,102 [I] DB FLAVOR :POSTGRES
2026-09-24 19:38:36,103 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:38:36,103 [I] Checking connection..
2026-09-24 19:38:36,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:38:36,360 [I] Checking connection passed.
2026-09-24 19:38:36,361 [I] --------- Verifying version history table ---------
2026-09-24 19:38:36,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:38:36,617 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:38:36,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:38:36,871 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:38:36,872 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:38:36,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:38:37.110 UTC [12289] WARNING: there is no transaction in progress
2026-09-24 19:38:37,116 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:38:37,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:38:37,347 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:38:37,347 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:38:37,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:38:37,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:38:37,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6kf3.c.gcp-upstream.internal','N') ;"
2026-09-24 19:38:38,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:38:38,299 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:38:38,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:38:38,526 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:38:38,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:38:38,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:38:38,996 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:38:38,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:38:39.922 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:39.938 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:39.954 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:39.969 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:39.983 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.001 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.055 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.061 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.076 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.086 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.096 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.109 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.119 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.125 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.131 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40.137 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 19:38:40,735 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:38:40,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:38:40,980 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:38:40,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:38:41,214 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:38:41,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:38:41,440 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:38:41,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:38:41,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:38:41,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6kf3.c.gcp-upstream.internal';"
2026-09-24 19:38:42,177 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:38:42,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:38:42,394 [I] DB_PATCHES have already been applied
I20260924 19:38:42.399413 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskm9q4Up/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:38:42.399472 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 19:38:42.402177 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41603
Sep 24, 2026 7:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:38:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:38:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:38:43.401893 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.996s user 0.003s sys 0.004s
Sep 24, 2026 7:39:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:39:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:39:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:39:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:39:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:39:04.296749 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:39:04 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:39:04.379454 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:39:04.388357 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:39:05.157505 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:39:05.157570 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms
I20260924 19:39:05.157582 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57213
2026-09-24 19:39:05,298 [I] DB FLAVOR :POSTGRES
2026-09-24 19:39:05,298 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:39:05,298 [I] Checking connection
2026-09-24 19:39:05,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:39:05,533 [I] connection success
2026-09-24 19:39:05,533 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:39:05,533 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:39:05,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:39:05,797 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:39:05,798 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:39:05,798 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:39:05,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:39:06,051 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:39:06.494727 392 mini_ranger_kms.cc:310] Created kms service
I20260924 19:39:06.688138 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:39:06.726388 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:39:06 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:07.100343 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:39:07.100546 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskm9q4Up/build/release/bin/postgresql.jar:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskm9q4Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:39:07.100569 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 19:39:07.103987 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12577
I20260924 19:39:07.104046 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45321
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:39:09.154963 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.776s user 0.001s sys 0.122s
I20260924 19:39:09.155047 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:39:09.155112 392 mini_ranger_kms.cc:401] 127.0.98.20:45321/kms/v1/keys
Sep 24 19:39:09 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:39:09 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:39:10 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:39:12 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:39:12.440032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32779
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:12.517381 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:12.517578 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:12.517598 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:12.518715 12628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:12.518756 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:12.518769 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:12.518780 12628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:12.518790 12628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:12.520408 12628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32779
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:12.520608 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:12.520823 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:39:12.526556 12628 server_base.cc:1034] running on GCE node
W20260924 19:39:12.526734 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:12.526713 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:12.526702 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:12.527144 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:12.527451 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:12.528590 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790278752528575 us; error 31 us; skew 500 ppm
Sep 24 19:39:12 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:12.534550 12628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:12.534987 12628 webserver.cc:460] Webserver started at http://127.0.98.62:44361/ using document root <none> and password file <none>
I20260924 19:39:12.535204 12628 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:12.535243 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:12.535347 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:39:12 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:12.586333 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "261ef5ff1afa459293c38f77e7e15379"
format_stamp: "Formatted at 2026-09-24 19:39:12 on dist-test-slave-6kf3"
server_key: "de9374a919ffd1943c75dc8bab4ec9b3"
server_key_iv: "7db0c557dbc3d1ff21ef8c87cbba4a76"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:12.586827 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "261ef5ff1afa459293c38f77e7e15379"
format_stamp: "Formatted at 2026-09-24 19:39:12 on dist-test-slave-6kf3"
server_key: "de9374a919ffd1943c75dc8bab4ec9b3"
server_key_iv: "7db0c557dbc3d1ff21ef8c87cbba4a76"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:12.588429 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:12.602366 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:12.602625 12628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:12.602756 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "261ef5ff1afa459293c38f77e7e15379"
format_stamp: "Formatted at 2026-09-24 19:39:12 on dist-test-slave-6kf3"
server_key: "de9374a919ffd1943c75dc8bab4ec9b3"
server_key_iv: "7db0c557dbc3d1ff21ef8c87cbba4a76"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:12.602866 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:12.612447 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:12.613777 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:12.613879 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:12.617530 12628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32779
I20260924 19:39:12.617565 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32779 every 8 connection(s)
I20260924 19:39:12.617864 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:12.618480 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:12.620659 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Bootstrap starting.
I20260924 19:39:12.621176 12696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:12.621417 12696 log.cc:824] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:12.622000 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: No bootstrap required, opened a new log
I20260924 19:39:12.623900 12696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:12.624078 12696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:12.624119 12696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261ef5ff1afa459293c38f77e7e15379, State: Initialized, Role: FOLLOWER
I20260924 19:39:12.624250 12696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:12.624306 12696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:12.624326 12696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:12.624370 12696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:12.624820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 12628
I20260924 19:39:12.624910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:39:12.624925 12696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:12.625022 12696 leader_election.cc:302] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261ef5ff1afa459293c38f77e7e15379; no voters:
I20260924 19:39:12.625089 392 external_mini_cluster.cc:1428] Setting key f4b95e8333d5fbbe165ff6a18164e399
I20260924 19:39:12.625181 12696 leader_election.cc:288] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:39:12.625206 12699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:12.625357 12699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 LEADER]: Becoming Leader. State: Replica: 261ef5ff1afa459293c38f77e7e15379, State: Running, Role: LEADER
I20260924 19:39:12.625413 12696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:12.625480 12699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:12.625875 12701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261ef5ff1afa459293c38f77e7e15379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260924 19:39:12.625855 12699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261ef5ff1afa459293c38f77e7e15379. Latest consensus state: current_term: 1 leader_uuid: "261ef5ff1afa459293c38f77e7e15379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260924 19:39:12.625936 12701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:12.625941 12699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:12.626448 12709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:12.626856 12696 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:39:12.627348 12709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:39:12 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:12.638295 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:12.625663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.0.98.62:32779)
0924 19:39:12.625815 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:12.625824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:12.625855 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:39:12.625982 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:12.625989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:12.626048 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:12.626186 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:12.626917 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:12.629865 (+ 2948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:12.630983 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:12.631767 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:12.631867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:12.634680 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:12.634685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:12.634685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:12.634698 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:12.636709 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:12.637074 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:12.637077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:12.637077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:12.637178 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:12.637363 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:12.637364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:12.637365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:12.637670 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:12.637828 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:12.637926 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:39:12.638053 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260924 19:39:12.640444 12709 catalog_manager.cc:1347] Generated new cluster ID: 896c3fe0792c43fbbc69077ecbf20851
I20260924 19:39:12.640496 12709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:12.649401 12709 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:39:12.650441 12709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:12.661832 12709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Generated new TSK 0
I20260924 19:39:12.662037 12709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:39:13.258410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:13.344712 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:13.344901 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:13.344921 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:13.346041 12741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:13.346084 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:13.346109 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:13.347541 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:13.347752 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:13.347999 12741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:13.353755 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:13.353823 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:13.353756 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:13.354321 12741 server_base.cc:1034] running on GCE node
I20260924 19:39:13.354514 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:13.354753 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:13.355891 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790278753355881 us; error 27 us; skew 500 ppm
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:13.358440 12741 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:13.358852 12741 webserver.cc:460] Webserver started at http://127.0.98.1:45291/ using document root <none> and password file <none>
I20260924 19:39:13.359066 12741 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:13.359113 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:13.359225 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:13.373234 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5894ebe66ea4fa2b14d50b302a14d46"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "04fab1ecea47b5b668deccac4d1821d9"
server_key_iv: "0a7caf330ea063877c5c71d19796a6e8"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.373692 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5894ebe66ea4fa2b14d50b302a14d46"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "04fab1ecea47b5b668deccac4d1821d9"
server_key_iv: "0a7caf330ea063877c5c71d19796a6e8"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.375451 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:39:13.388262 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.388500 12741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:13.388622 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "c5894ebe66ea4fa2b14d50b302a14d46"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "04fab1ecea47b5b668deccac4d1821d9"
server_key_iv: "0a7caf330ea063877c5c71d19796a6e8"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.388734 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:13.431269 12741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:13.433346 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:13.433477 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:13.433735 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:13.434068 12741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:13.434104 12741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.434132 12741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:13.434170 12741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.440702 12741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37571
I20260924 19:39:13.441207 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:39:13.441782 12871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37571 every 8 connection(s)
I20260924 19:39:13.443950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 12741
I20260924 19:39:13.444049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:39:13.444257 392 external_mini_cluster.cc:1428] Setting key 2ed09bc6c06d9f9c42f4e68667320bf3
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 19:39:13.465183 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.443034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41305 (local address 127.0.98.62:32779)
0924 19:39:13.449021 (+ 5987us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:13.449023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:13.449031 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:39:13.449068 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:13.449069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:13.449082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:13.449151 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:13.449971 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.450751 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.452269 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.452586 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.452648 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:13.461540 (+ 8892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:13.461542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:13.461542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:13.461551 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:13.463301 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:13.463887 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:13.463889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:13.463890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:13.463984 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:13.464318 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:13.464319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:13.464320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:13.464466 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:13.464517 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:13.464941 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 19:39:13.465036 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5954,"thread_start_us":67,"threads_started":1}
I20260924 19:39:13.466010 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.442789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.1:41305)
0924 19:39:13.448097 (+ 5308us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:13.448159 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:13.448814 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:13.449300 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:13.449305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:13.449556 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:13.449855 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:13.449864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.450855 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.450856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:13.452164 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:13.452167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.452733 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.452734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:13.452875 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:13.453447 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:13.453458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:13.461396 (+ 7938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:13.463407 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:13.463420 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:13.463423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:13.463784 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:13.464066 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:13.464067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:13.464067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:13.464235 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:13.464553 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:13.464563 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:13.464768 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 19:39:13.465548 (+ 780us) client_negotiation.cc:240] Negotiation successful
0924 19:39:13.465761 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5239,"thread_start_us":84,"threads_started":1}
I20260924 19:39:13.466758 12872 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:13.466854 12872 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:13.467042 12872 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:13.467656 12660 ts_manager.cc:194] Registered new tserver with Master: c5894ebe66ea4fa2b14d50b302a14d46 (127.0.98.1:37571)
I20260924 19:39:13.468561 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:39:13.484891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:13.594651 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:13.594935 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:13.594967 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:13.596705 12878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:13.596814 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:13.596872 12878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:13.599220 12878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:13.599637 12878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:13.599922 12878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:13.606091 12888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:13.606097 12886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:13.606521 12878 server_base.cc:1034] running on GCE node
W20260924 19:39:13.606359 12885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:13.606809 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:13.607089 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:13.608251 12878 hybrid_clock.cc:648] HybridClock initialized: now 1790278753608224 us; error 42 us; skew 500 ppm
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:13.611369 12878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:13.611913 12878 webserver.cc:460] Webserver started at http://127.0.98.2:41347/ using document root <none> and password file <none>
I20260924 19:39:13.612246 12878 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:13.612327 12878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:13.612469 12878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:13.641827 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c57605df4c74a9ea686d77c53fdb092"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "d75c2c62732acb4ddcb3697f11833c7b"
server_key_iv: "54d83c7e9919c7a60a9439eed4c57c44"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.642311 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c57605df4c74a9ea686d77c53fdb092"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "d75c2c62732acb4ddcb3697f11833c7b"
server_key_iv: "54d83c7e9919c7a60a9439eed4c57c44"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.644649 12878 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260924 19:39:13.664062 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.664275 12878 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 19:39:13.664383 12878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "0c57605df4c74a9ea686d77c53fdb092"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "d75c2c62732acb4ddcb3697f11833c7b"
server_key_iv: "54d83c7e9919c7a60a9439eed4c57c44"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.664492 12878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:13.680612 12878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:13.682960 12878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:13.683183 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:13.683490 12878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:13.683900 12878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:13.684021 12878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.684095 12878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:13.684150 12878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.698706 12878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39979
I20260924 19:39:13.699236 12878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:39:13.704218 13009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39979 every 8 connection(s)
I20260924 19:39:13.707463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 12878
I20260924 19:39:13.707569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:39:13.707752 392 external_mini_cluster.cc:1428] Setting key fd7606485900e167f69943553ba91651
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:13.729199 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.706067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34247 (local address 127.0.98.62:32779)
0924 19:39:13.706140 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:13.706142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:13.707475 (+ 1333us) server_negotiation.cc:407] Connection header received
0924 19:39:13.708567 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:13.708568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:13.708579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:13.708617 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:13.709514 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.710286 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.720486 (+ 10200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.720844 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.720903 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:13.725835 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:13.725837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:13.725837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:13.725846 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:13.727500 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:13.728050 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:13.728051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:13.728051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:13.728112 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:13.728398 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:13.728399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:13.728399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:13.728550 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:13.728603 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:13.728998 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 19:39:13.729096 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:39:13.730000 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.705713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.2:34247)
0924 19:39:13.707325 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:13.707381 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:13.708066 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:13.708830 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:13.708836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:13.709089 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:13.709399 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:13.709407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.710399 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.710400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:13.720192 (+ 9792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:13.720198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.721061 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.721062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:13.721334 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:13.724010 (+ 2676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:13.724022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:13.725707 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:13.727617 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:13.727626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:13.727629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:13.727959 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:13.728176 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:13.728177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:13.728178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:13.728324 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:13.728638 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:13.728645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:13.728839 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 19:39:13.729556 (+ 717us) client_negotiation.cc:240] Negotiation successful
0924 19:39:13.729742 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"thread_start_us":116,"threads_started":1}
I20260924 19:39:13.730855 13010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:13.730950 13010 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:13.731148 13010 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:13.731638 12660 ts_manager.cc:194] Registered new tserver with Master: 0c57605df4c74a9ea686d77c53fdb092 (127.0.98.2:39979)
I20260924 19:39:13.732199 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:39:13.746444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:13.864127 13015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:13.864444 13015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:13.864535 13015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:13.866380 13015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:13.866485 13015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:13.866595 13015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:13.868661 13015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:13.869082 13015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:13.869362 13015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:13.875358 13023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:13.880749 13020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:13.881309 13015 server_base.cc:1034] running on GCE node
W20260924 19:39:13.884094 13021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:13.884436 13015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:13.884790 13015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:13.900415 13015 hybrid_clock.cc:648] HybridClock initialized: now 1790278753900386 us; error 49 us; skew 500 ppm
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:13.904201 13015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:13.904743 13015 webserver.cc:460] Webserver started at http://127.0.98.3:44657/ using document root <none> and password file <none>
I20260924 19:39:13.904994 13015 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:13.905042 13015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:13.905164 13015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:13.920132 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "ced93713d69647659a0f4970ee2a0347"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "0bb6c509d2a7d33882bb4a2d8e376f09"
server_key_iv: "2b8b13029044751fa15909a89f1df2b0"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.920631 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ced93713d69647659a0f4970ee2a0347"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "0bb6c509d2a7d33882bb4a2d8e376f09"
server_key_iv: "2b8b13029044751fa15909a89f1df2b0"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.922708 13015 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:13.940164 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.940388 13015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260924 19:39:13.940506 13015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "ced93713d69647659a0f4970ee2a0347"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "0bb6c509d2a7d33882bb4a2d8e376f09"
server_key_iv: "2b8b13029044751fa15909a89f1df2b0"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:13.940632 13015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:13.975456 13015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:13.977905 13015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:13.978120 13015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:13.978420 13015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:13.978840 13015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:13.978917 13015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.978979 13015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:13.979031 13015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:13.991792 13015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44159
I20260924 19:39:13.992324 13015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:39:13.995282 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44159 every 8 connection(s)
I20260924 19:39:13.995882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13015
I20260924 19:39:13.995995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:39:13.996208 392 external_mini_cluster.cc:1428] Setting key 219cef23f88df912a8916007a41d4523
Sep 24 19:39:13 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:14.007561 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.996596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33759 (local address 127.0.98.62:32779)
0924 19:39:13.996641 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:13.996643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:13.997014 (+ 371us) server_negotiation.cc:407] Connection header received
0924 19:39:13.997812 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:13.997813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:13.997826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:13.997852 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:13.998694 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.999503 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.001039 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.001350 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.001391 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.003963 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:14.003964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:14.003965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:14.003972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:14.005780 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.006262 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.006263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.006264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.006323 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.006610 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.006611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.006611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.006764 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:14.006814 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:14.007375 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 19:39:14.007462 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:39:14.008610 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:13.996523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.3:33759)
0924 19:39:13.996891 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:13.996941 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:13.997694 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:13.998038 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:13.998043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:13.998301 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:13.998588 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:13.998594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:13.999624 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:13.999624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.000938 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.000941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.001488 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.001489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.001644 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.002192 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:14.002202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:14.003839 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:14.005877 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.005884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.005887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.006173 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.006388 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.006388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.006389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.006521 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.006874 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:14.006881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:14.007060 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 19:39:14.008093 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0924 19:39:14.008354 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":85,"threads_started":1}
I20260924 19:39:14.009222 13144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:14.009311 13144 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.009487 13144 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:14.009876 12660 ts_manager.cc:194] Registered new tserver with Master: ced93713d69647659a0f4970ee2a0347 (127.0.98.3:44159)
I20260924 19:39:14.010346 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33759
I20260924 19:39:14.019069 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:39:14.020147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 12741
I20260924 19:39:14.026289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 12878
I20260924 19:39:14.031253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13015
I20260924 19:39:14.036255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 12628
I20260924 19:39:14.042102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--webserver_interface=127.0.98.62
--webserver_port=44361
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32779
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:14.122897 13147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:14.123095 13147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:14.123116 13147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:14.124264 13147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:14.124303 13147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:14.124316 13147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:14.124339 13147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:14.124362 13147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:14.125926 13147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32779
--ranger_config_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:14.126142 13147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:14.126341 13147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:14.130833 13152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.130826 13155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.130824 13153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:14.131138 13147 server_base.cc:1034] running on GCE node
I20260924 19:39:14.131382 13147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:14.131670 13147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:14.132828 13147 hybrid_clock.cc:648] HybridClock initialized: now 1790278754132814 us; error 23 us; skew 500 ppm
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:14.135370 13147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:14.135782 13147 webserver.cc:460] Webserver started at http://127.0.98.62:44361/ using document root <none> and password file <none>
I20260924 19:39:14.136031 13147 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:14.136087 13147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:14.179178 13147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:14.180064 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.180270 13147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:14.180367 13147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "261ef5ff1afa459293c38f77e7e15379"
format_stamp: "Formatted at 2026-09-24 19:39:12 on dist-test-slave-6kf3"
server_key: "de9374a919ffd1943c75dc8bab4ec9b3"
server_key_iv: "7db0c557dbc3d1ff21ef8c87cbba4a76"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:14.180737 13147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:14.198941 13147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:14.200486 13147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:14.200608 13147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:14.204362 13147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32779
I20260924 19:39:14.204412 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32779 every 8 connection(s)
I20260924 19:39:14.204733 13147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:14.205862 13216 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:39:14.206465 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Bootstrap starting.
I20260924 19:39:14.208077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13147
I20260924 19:39:14.208359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37571
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32779
--webserver_port=45291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:14.214228 13216 log.cc:824] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:14.215799 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:39:14.216063 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Bootstrap complete.
I20260924 19:39:14.218142 13216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:14.218372 13216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261ef5ff1afa459293c38f77e7e15379, State: Initialized, Role: FOLLOWER
I20260924 19:39:14.218508 13216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:14.218569 13216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:14.218596 13216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:14.218638 13216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:39:14.219554 13216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:14.219642 13216 leader_election.cc:302] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261ef5ff1afa459293c38f77e7e15379; no voters:
I20260924 19:39:14.219784 13216 leader_election.cc:288] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:39:14.219827 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:39:14.220000 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [term 2 LEADER]: Becoming Leader. State: Replica: 261ef5ff1afa459293c38f77e7e15379, State: Running, Role: LEADER
I20260924 19:39:14.220055 13216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:14.220088 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260924 19:39:14.220413 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "261ef5ff1afa459293c38f77e7e15379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260924 19:39:14.220431 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261ef5ff1afa459293c38f77e7e15379. Latest consensus state: current_term: 2 leader_uuid: "261ef5ff1afa459293c38f77e7e15379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ef5ff1afa459293c38f77e7e15379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260924 19:39:14.220659 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:14.220844 13229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:14.220569 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:14.221304 13229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:14.221474 13229 catalog_manager.cc:1259] Loaded cluster ID: 896c3fe0792c43fbbc69077ecbf20851
I20260924 19:39:14.221536 13229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:14.221621 13216 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:39:14.222738 13229 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:14.222929 13229 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Loaded TSK: 0
I20260924 19:39:14.223131 13229 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:39:14.301031 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:14.301234 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:14.301265 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:14.302613 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:14.302654 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:14.302688 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:14.304314 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45291
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:14.304541 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:14.304769 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:14.309479 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.309746 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.309962 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:14.310047 13218 server_base.cc:1034] running on GCE node
I20260924 19:39:14.310300 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:14.310523 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:14.311678 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790278754311649 us; error 24 us; skew 500 ppm
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:14.314244 13218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:14.314652 13218 webserver.cc:460] Webserver started at http://127.0.98.1:45291/ using document root <none> and password file <none>
I20260924 19:39:14.314893 13218 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:14.314953 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:14.336700 13218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:14.340039 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.340201 13218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:39:14.340286 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "c5894ebe66ea4fa2b14d50b302a14d46"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "04fab1ecea47b5b668deccac4d1821d9"
server_key_iv: "0a7caf330ea063877c5c71d19796a6e8"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:14.340636 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:14.369835 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:14.371659 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:14.371799 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:14.372056 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:14.372424 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:14.372462 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.372490 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:14.372529 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.378585 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37571
I20260924 19:39:14.379103 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:39:14.380509 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37571 every 8 connection(s)
I20260924 19:39:14.383641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13218
I20260924 19:39:14.383911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32779
--webserver_port=41347
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:14.403800 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.382419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.1:52511)
0924 19:39:14.382578 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:14.382622 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:14.383206 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:14.392686 (+ 9480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:14.392692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:14.392947 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:14.393270 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.393279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.395237 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.395239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.396568 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.396572 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.397405 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.397406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.397550 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.398132 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:14.398142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:14.399736 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:14.401991 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.401999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.402001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.402302 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.402567 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.402567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.402568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.402689 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.403127 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:14.403133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:14.403308 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 19:39:14.403415 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:39:14.403562 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260924 19:39:14.403805 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.382522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52511 (local address 127.0.98.62:32779)
0924 19:39:14.392085 (+ 9563us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:14.392098 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:14.392150 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:39:14.392337 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:14.392347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:14.392416 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:14.393416 (+ 1000us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:14.393520 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.395121 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.396657 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.397300 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.397416 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.399850 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:14.399855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:14.399855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:14.399867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:14.401915 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.402384 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.402387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.402388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.402497 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.402751 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.402752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.402753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.403057 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:14.403144 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:14.403476 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 19:39:14.403602 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9483,"thread_start_us":86,"threads_started":1}
I20260924 19:39:14.405079 13378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:14.405180 13378 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.405354 13378 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.406220 13378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32779 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:39:14.406289 13378 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.406350 13378 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:14.406724 13378 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.406802 13378 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.407083 13378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:39:14.490609 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:14.490964 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:14.491060 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:14.492815 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:14.492919 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:14.493000 13382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:14.495112 13382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41347
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:14.495532 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:14.495824 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:14.501891 13390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.501873 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.502517 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:14.503161 13382 server_base.cc:1034] running on GCE node
I20260924 19:39:14.503474 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:14.503746 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:14.504911 13382 hybrid_clock.cc:648] HybridClock initialized: now 1790278754504882 us; error 42 us; skew 500 ppm
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:14.508059 13382 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:14.508638 13382 webserver.cc:460] Webserver started at http://127.0.98.2:41347/ using document root <none> and password file <none>
I20260924 19:39:14.508913 13382 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:14.508998 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:14.535733 13382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:14.536566 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.536725 13382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.536810 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "0c57605df4c74a9ea686d77c53fdb092"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "d75c2c62732acb4ddcb3697f11833c7b"
server_key_iv: "54d83c7e9919c7a60a9439eed4c57c44"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:14.537247 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:14.564147 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:14.566352 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:14.566607 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:14.566885 13382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:14.567317 13382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:14.567404 13382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.567461 13382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:14.567504 13382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.573971 13382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39979
I20260924 19:39:14.574057 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39979 every 8 connection(s)
I20260924 19:39:14.574434 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:14.581378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13382
I20260924 19:39:14.581624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44159
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32779
--webserver_port=44657
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:14.585407 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.575138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.2:60049)
0924 19:39:14.575298 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:14.575338 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:14.575897 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:14.576257 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:14.576262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:14.576429 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:14.576658 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.576663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.577632 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.577633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.578614 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.578617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.578986 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.578987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.579101 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.579545 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:14.579553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:14.580778 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:14.583916 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.583921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.583924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.584195 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.584378 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.584379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.584379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.584495 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.584779 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:14.584784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:14.584948 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 19:39:14.585039 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:39:14.585181 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260924 19:39:14.586760 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.575162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60049 (local address 127.0.98.62:32779)
0924 19:39:14.575234 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:14.575236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:14.575433 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:39:14.576069 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:14.576071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:14.576090 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:14.576178 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:14.576762 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.577537 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.578698 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.578907 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.578960 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.580861 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:14.580862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:14.580863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:14.580870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:14.583830 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.584255 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.584257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.584257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.584314 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.584573 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.584574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.584574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.584696 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:14.584757 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:14.586580 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0924 19:39:14.586669 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:39:14.587317 13512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:14.587410 13512 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.587574 13512 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.588191 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32779 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:39:14.588256 13512 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.588318 13512 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:14.588625 13512 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.588706 13512 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.588987 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:39:14.684888 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:14.685184 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:14.685223 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:14.686980 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:14.687090 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:14.687160 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:14.689373 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44657
--tserver_master_addrs=127.0.98.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:14.689759 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:14.690045 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:14.696213 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.696244 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:14.696570 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:14.697180 13515 server_base.cc:1034] running on GCE node
I20260924 19:39:14.697428 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:14.697736 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:14.698910 13515 hybrid_clock.cc:648] HybridClock initialized: now 1790278754698884 us; error 37 us; skew 500 ppm
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:14.702178 13515 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:14.702728 13515 webserver.cc:460] Webserver started at http://127.0.98.3:44657/ using document root <none> and password file <none>
I20260924 19:39:14.702996 13515 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:14.703132 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:14.727743 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:39:14.728668 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.728868 13515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:14.728969 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "ced93713d69647659a0f4970ee2a0347"
format_stamp: "Formatted at 2026-09-24 19:39:13 on dist-test-slave-6kf3"
server_key: "0bb6c509d2a7d33882bb4a2d8e376f09"
server_key_iv: "2b8b13029044751fa15909a89f1df2b0"
server_key_version: "kuduclusterkey@0"
I20260924 19:39:14.729475 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:14.749826 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:14.751917 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:14.752118 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:14.752358 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:14.752707 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:14.752745 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.752772 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:14.752799 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:14.759312 13515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44159
I20260924 19:39:14.759383 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44159 every 8 connection(s)
I20260924 19:39:14.759793 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:39:14.762511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13515
Sep 24 19:39:14 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:14.771242 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.760553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57979 (local address 127.0.98.62:32779)
0924 19:39:14.760604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:14.760605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:14.760938 (+ 333us) server_negotiation.cc:407] Connection header received
0924 19:39:14.761581 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:14.761583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:14.761595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:14.761672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:14.762726 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.763363 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.764805 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.765097 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.765141 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.768323 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:14.768324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:14.768325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:14.768333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:14.769895 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.770295 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.770296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.770297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.770358 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.770608 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.770609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.770609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.770723 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:14.770784 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:14.771069 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 19:39:14.771147 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:39:14.771410 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.760520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.3:57979)
0924 19:39:14.760791 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:14.760833 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:14.761445 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:14.762119 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:14.762124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:14.762350 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:14.762625 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.762632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.763455 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.763456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.764701 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:14.764704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.765256 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.765257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:14.765399 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.765937 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:14.765946 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:14.768222 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:14.769984 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.769991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.769994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.770225 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.770416 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:14.770417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:14.770417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:14.770547 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:14.770781 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:14.770787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:14.770937 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:39:14.771042 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:39:14.771186 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":88,"threads_started":1}
I20260924 19:39:14.772014 13646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260924 19:39:14.772117 13646 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.772300 13646 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.773070 13646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32779 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:39:14.773149 13646 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.773216 13646 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:14.773613 13646 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:14.773687 13646 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
W20260924 19:39:14.774031 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:39:14.777149 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:14.763014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.98.62:32779)
0924 19:39:14.763462 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:14.763464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:14.763469 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:39:14.763484 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:14.763484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:14.763493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:14.763586 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:14.765746 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.767458 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.768898 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:14.769164 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:14.769230 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:14.774617 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:14.774618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:14.774618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:14.774625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:14.776059 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.776452 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.776453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.776453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.776506 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:14.776700 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:14.776701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:14.776701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:14.776794 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:14.776917 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:14.777013 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:39:14.777079 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":53,"threads_started":1}
Sep 24 19:39:15 dist-test-slave-6kf3 krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:15.407279 13378 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:15.407449 13378 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:15.407989 13180 ts_manager.cc:194] Registered new tserver with Master: c5894ebe66ea4fa2b14d50b302a14d46 (127.0.98.1:37571)
I20260924 19:39:15.408912 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52511
I20260924 19:39:15.589159 13512 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:15.589308 13512 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:15.589629 13180 ts_manager.cc:194] Registered new tserver with Master: 0c57605df4c74a9ea686d77c53fdb092 (127.0.98.2:39979)
I20260924 19:39:15.590147 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60049
Sep 24 19:39:15 dist-test-slave-6kf3 krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:39:15.774224 13646 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:15.774369 13646 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260924 19:39:15.774716 13180 ts_manager.cc:194] Registered new tserver with Master: ced93713d69647659a0f4970ee2a0347 (127.0.98.3:44159)
I20260924 19:39:15.775213 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57979
I20260924 19:39:15.782655 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:39:15.791761 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.783720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.62:32779)
0924 19:39:15.784021 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.784023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.784179 (+ 156us) server_negotiation.cc:407] Connection header received
0924 19:39:15.784201 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.784201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.784212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.784264 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:15.784912 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.785676 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.786661 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.786962 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.787030 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.788527 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:15.788529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:15.788529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:15.788536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:15.790266 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:15.790643 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:15.790644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:15.790645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:15.790707 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:15.790892 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:15.790893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:15.790893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:15.791038 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:15.791135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.791572 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.791662 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":63,"threads_started":1}
I20260924 19:39:15.794284 13180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:39:15.796759 13180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:39:15.810106 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.800822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44159 (local address 127.0.0.1:57216)
0924 19:39:15.802384 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.802392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:15.802432 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:15.802715 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:15.802718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:15.802720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:15.802791 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.802793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.804345 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.804346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.808753 (+ 4407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.808755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.809801 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.809802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.809888 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.809901 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:39:15.809966 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:39:15.810017 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"thread_start_us":93,"threads_started":1}
I20260924 19:39:15.810568 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.801063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.0.98.3:44159)
0924 19:39:15.802094 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.802100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.802484 (+ 384us) server_negotiation.cc:407] Connection header received
0924 19:39:15.802539 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.802541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.802557 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.802624 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:15.802859 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.804249 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.808839 (+ 4590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.809611 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.809689 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.809765 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.810432 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.810477 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":42,"threads_started":1}
I20260924 19:39:15.812016 13580 tablet_service.cc:1467] Processing CreateTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd (DEFAULT_TABLE table=test-table [id=2dc9a5292ed645dd8df21b50c3fb5782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:15.812402 13580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bc2ce5e22f4df49fa15a9789a22bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:15.813118 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.801019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39979 (local address 127.0.0.1:52904)
0924 19:39:15.805857 (+ 4838us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.805868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:15.805923 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:15.806333 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:15.806334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:15.806336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:15.806374 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.806375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.807579 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.807580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.811178 (+ 3598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.811181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.812881 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.812882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.812943 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.812947 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:39:15.812998 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:39:15.813039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4823,"thread_start_us":61,"threads_started":1}
I20260924 19:39:15.814587 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.801210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.98.2:39979)
0924 19:39:15.806060 (+ 4850us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.806068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.806082 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:39:15.806126 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.806129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.806150 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.806245 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:15.806436 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.807473 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.811964 (+ 4491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.812708 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.812779 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.812848 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.814453 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.814499 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4804,"thread_start_us":54,"threads_started":1}
I20260924 19:39:15.815189 13446 tablet_service.cc:1467] Processing CreateTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd (DEFAULT_TABLE table=test-table [id=2dc9a5292ed645dd8df21b50c3fb5782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:15.815497 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bc2ce5e22f4df49fa15a9789a22bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:15.822866 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.800815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.0.1:49490)
0924 19:39:15.802172 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.802216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:15.802284 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:15.811586 (+ 9302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:15.811588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:15.811590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:15.811627 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.811628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.820280 (+ 8652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.820282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.821394 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.821398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.822664 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.822665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.822728 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.822733 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:39:15.822758 (+ 25us) client_negotiation.cc:240] Negotiation successful
0924 19:39:15.822797 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":75,"threads_started":1}
I20260924 19:39:15.823241 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.800904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.0.98.1:37571)
0924 19:39:15.811356 (+ 10452us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.811361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.811372 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:39:15.811405 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.811407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.811421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.811489 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:15.813264 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.814332 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.821505 (+ 7173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.822574 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.823025 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.823097 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.823123 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.823165 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10414,"thread_start_us":44,"threads_started":1}
I20260924 19:39:15.823271 13680 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Bootstrap starting.
I20260924 19:39:15.823522 13681 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Bootstrap starting.
I20260924 19:39:15.823971 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd (DEFAULT_TABLE table=test-table [id=2dc9a5292ed645dd8df21b50c3fb5782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:15.824226 13681 tablet_bootstrap.cc:654] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:15.824294 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bc2ce5e22f4df49fa15a9789a22bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:15.824520 13681 log.cc:824] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:15.825248 13681 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: No bootstrap required, opened a new log
I20260924 19:39:15.825309 13681 ts_tablet_manager.cc:1397] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:15.827978 13681 raft_consensus.cc:348] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.828181 13681 raft_consensus.cc:374] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:15.828243 13681 raft_consensus.cc:729] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c57605df4c74a9ea686d77c53fdb092, State: Initialized, Role: FOLLOWER
I20260924 19:39:15.828404 13681 consensus_queue.cc:260] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.828595 13680 tablet_bootstrap.cc:654] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:15.828933 13680 log.cc:824] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:15.829013 13681 ts_tablet_manager.cc:1428] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 19:39:15.829154 13512 heartbeater.cc:503] Master 127.0.98.62:32779 was elected leader, sending a full tablet report...
I20260924 19:39:15.830856 13680 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: No bootstrap required, opened a new log
I20260924 19:39:15.830919 13680 ts_tablet_manager.cc:1397] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:39:15.833405 13680 raft_consensus.cc:348] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.833546 13680 raft_consensus.cc:374] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:15.833575 13680 raft_consensus.cc:729] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced93713d69647659a0f4970ee2a0347, State: Initialized, Role: FOLLOWER
I20260924 19:39:15.833815 13680 consensus_queue.cc:260] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.834152 13680 ts_tablet_manager.cc:1428] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 19:39:15.834290 13646 heartbeater.cc:503] Master 127.0.98.62:32779 was elected leader, sending a full tablet report...
I20260924 19:39:15.835433 13683 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Bootstrap starting.
I20260924 19:39:15.836171 13683 tablet_bootstrap.cc:654] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:15.837239 13683 log.cc:824] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:15.837983 13683 tablet_bootstrap.cc:492] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: No bootstrap required, opened a new log
I20260924 19:39:15.838053 13683 ts_tablet_manager.cc:1397] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:39:15.840572 13683 raft_consensus.cc:348] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.840732 13683 raft_consensus.cc:374] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:15.840767 13683 raft_consensus.cc:729] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5894ebe66ea4fa2b14d50b302a14d46, State: Initialized, Role: FOLLOWER
I20260924 19:39:15.841017 13683 consensus_queue.cc:260] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.841395 13683 ts_tablet_manager.cc:1428] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:39:15.841549 13378 heartbeater.cc:503] Master 127.0.98.62:32779 was elected leader, sending a full tablet report...
W20260924 19:39:15.882046 13379 tablet.cc:2369] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:39:15.889122 13686 raft_consensus.cc:482] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:15.889267 13686 raft_consensus.cc:504] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.889649 13686 leader_election.cc:288] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5894ebe66ea4fa2b14d50b302a14d46 (127.0.98.1:37571), 0c57605df4c74a9ea686d77c53fdb092 (127.0.98.2:39979)
I20260924 19:39:15.893501 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.890195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45661 (local address 127.0.98.2:39979)
0924 19:39:15.890233 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.890235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.890324 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:39:15.890405 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.890406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.890417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.890444 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:15.890682 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.891339 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.892608 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.893153 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.893191 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.893203 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.893392 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.893430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:39:15.893913 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.889901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58477 (local address 127.0.98.1:37571)
0924 19:39:15.889945 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.889946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.890073 (+ 127us) server_negotiation.cc:407] Connection header received
0924 19:39:15.890173 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.890174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.890186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.890214 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:15.890447 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.891172 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.893029 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.893557 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.893599 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.893608 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.893810 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.893850 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:39:15.894016 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.890032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39979 (local address 127.0.98.3:45661)
0924 19:39:15.890276 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.890284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:15.890343 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:15.890498 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:15.890498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:15.890499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:15.890596 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.890604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.891413 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.891414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.892505 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.892508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.893257 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.893258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.893325 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.893328 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:15.893897 (+ 569us) client_negotiation.cc:240] Negotiation successful
0924 19:39:15.893940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260924 19:39:15.894294 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bc2ce5e22f4df49fa15a9789a22bfd" candidate_uuid: "ced93713d69647659a0f4970ee2a0347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c57605df4c74a9ea686d77c53fdb092" is_pre_election: true
I20260924 19:39:15.894367 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.889833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.98.3:58477)
0924 19:39:15.889999 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.890020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:15.890100 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:15.890283 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:15.890284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:15.890286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:15.890377 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.890379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.891260 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.891261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.892931 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:15.892934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.893676 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.893677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:15.893740 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.893743 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:15.894237 (+ 494us) client_negotiation.cc:240] Negotiation successful
0924 19:39:15.894278 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":8,"thread_start_us":92,"threads_started":1}
I20260924 19:39:15.894466 13466 raft_consensus.cc:2454] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced93713d69647659a0f4970ee2a0347 in term 0.
I20260924 19:39:15.894651 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bc2ce5e22f4df49fa15a9789a22bfd" candidate_uuid: "ced93713d69647659a0f4970ee2a0347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" is_pre_election: true
I20260924 19:39:15.894714 13535 leader_election.cc:302] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c57605df4c74a9ea686d77c53fdb092, ced93713d69647659a0f4970ee2a0347; no voters:
I20260924 19:39:15.894793 13327 raft_consensus.cc:2454] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced93713d69647659a0f4970ee2a0347 in term 0.
I20260924 19:39:15.894840 13686 raft_consensus.cc:2781] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:39:15.894881 13686 raft_consensus.cc:482] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:39:15.894904 13686 raft_consensus.cc:3037] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:15.895682 13686 raft_consensus.cc:504] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.895790 13686 leader_election.cc:288] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [CANDIDATE]: Term 1 election: Requested vote from peers c5894ebe66ea4fa2b14d50b302a14d46 (127.0.98.1:37571), 0c57605df4c74a9ea686d77c53fdb092 (127.0.98.2:39979)
I20260924 19:39:15.896006 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bc2ce5e22f4df49fa15a9789a22bfd" candidate_uuid: "ced93713d69647659a0f4970ee2a0347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5894ebe66ea4fa2b14d50b302a14d46"
I20260924 19:39:15.896019 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bc2ce5e22f4df49fa15a9789a22bfd" candidate_uuid: "ced93713d69647659a0f4970ee2a0347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c57605df4c74a9ea686d77c53fdb092"
I20260924 19:39:15.896086 13466 raft_consensus.cc:3037] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:15.896086 13327 raft_consensus.cc:3037] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:15.896893 13466 raft_consensus.cc:2454] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced93713d69647659a0f4970ee2a0347 in term 1.
I20260924 19:39:15.896893 13327 raft_consensus.cc:2454] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced93713d69647659a0f4970ee2a0347 in term 1.
I20260924 19:39:15.897074 13535 leader_election.cc:302] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c57605df4c74a9ea686d77c53fdb092, ced93713d69647659a0f4970ee2a0347; no voters:
I20260924 19:39:15.897163 13686 raft_consensus.cc:2781] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:15.897296 13686 raft_consensus.cc:686] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 1 LEADER]: Becoming Leader. State: Replica: ced93713d69647659a0f4970ee2a0347, State: Running, Role: LEADER
I20260924 19:39:15.897378 13686 consensus_queue.cc:237] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } }
I20260924 19:39:15.898166 13180 catalog_manager.cc:5495] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced93713d69647659a0f4970ee2a0347 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ced93713d69647659a0f4970ee2a0347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced93713d69647659a0f4970ee2a0347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 } health_report { overall_health: UNKNOWN } } }
I20260924 19:39:15.921226 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.918094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.0.98.3:44159)
0924 19:39:15.918165 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:15.918167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:15.918177 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:39:15.918220 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:15.918221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:15.918247 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:15.918342 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:39:15.918473 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.919180 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.920274 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:15.920549 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:15.920627 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:15.920836 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:39:15.920962 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:39:15.921029 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:15.921106 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:39:15.921131 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:39:15.925166 13327 raft_consensus.cc:1261] T 63bc2ce5e22f4df49fa15a9789a22bfd P c5894ebe66ea4fa2b14d50b302a14d46 [term 1 FOLLOWER]: Refusing update from remote peer ced93713d69647659a0f4970ee2a0347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:15.925190 13466 raft_consensus.cc:1261] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 1 FOLLOWER]: Refusing update from remote peer ced93713d69647659a0f4970ee2a0347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:15.925544 13686 consensus_queue.cc:1035] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c57605df4c74a9ea686d77c53fdb092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:15.925719 13686 consensus_queue.cc:1035] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5894ebe66ea4fa2b14d50b302a14d46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:15.929214 13694 mvcc.cc:205] Tried to move back new op lower bound from 7332981784263471104 to 7332981784155992064. Current Snapshot: MvccSnapshot[applied={T|T < 7332981784263471104}]
I20260924 19:39:15.929905 13696 mvcc.cc:205] Tried to move back new op lower bound from 7332981784263471104 to 7332981784155992064. Current Snapshot: MvccSnapshot[applied={T|T < 7332981784263471104}]
I20260924 19:39:15.931749 13695 mvcc.cc:205] Tried to move back new op lower bound from 7332981784263471104 to 7332981784155992064. Current Snapshot: MvccSnapshot[applied={T|T < 7332981784263471104}]
I20260924 19:39:15.934777 13179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:15.934864 13179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:15.935612 13179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 261ef5ff1afa459293c38f77e7e15379: Sending DeleteTablet for 3 replicas of tablet 63bc2ce5e22f4df49fa15a9789a22bfd
I20260924 19:39:15.936103 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49490
I20260924 19:39:15.936321 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52904
I20260924 19:39:15.936537 13580 tablet_service.cc:1514] Processing DeleteTablet for tablet 63bc2ce5e22f4df49fa15a9789a22bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57216
I20260924 19:39:15.937181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13218
I20260924 19:39:15.942243 13705 tablet_replica.cc:330] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: stopping tablet replica
I20260924 19:39:15.942391 13706 tablet_replica.cc:330] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: stopping tablet replica
I20260924 19:39:15.942538 13706 raft_consensus.cc:2229] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:39:15.942687 13705 raft_consensus.cc:2229] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:15.942854 13706 raft_consensus.cc:2258] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:15.942989 13705 raft_consensus.cc:2258] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:15.944401 13706 ts_tablet_manager.cc:1905] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:15.944752 13705 ts_tablet_manager.cc:1905] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:15.946086 13706 ts_tablet_manager.cc:1918] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:15.946352 13705 ts_tablet_manager.cc:1918] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:15.947827 13706 log.cc:1196] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/63bc2ce5e22f4df49fa15a9789a22bfd
I20260924 19:39:15.948128 13706 ts_tablet_manager.cc:1939] T 63bc2ce5e22f4df49fa15a9789a22bfd P ced93713d69647659a0f4970ee2a0347: Deleting consensus metadata
I20260924 19:39:15.948582 13164 catalog_manager.cc:4841] TS ced93713d69647659a0f4970ee2a0347 (127.0.98.3:44159): tablet 63bc2ce5e22f4df49fa15a9789a22bfd (table test-table [id=2dc9a5292ed645dd8df21b50c3fb5782]) successfully deleted
W20260924 19:39:15.948698 13166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37571 (error 108)
I20260924 19:39:15.948786 13705 log.cc:1196] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/63bc2ce5e22f4df49fa15a9789a22bfd
I20260924 19:39:15.949033 13705 ts_tablet_manager.cc:1939] T 63bc2ce5e22f4df49fa15a9789a22bfd P 0c57605df4c74a9ea686d77c53fdb092: Deleting consensus metadata
I20260924 19:39:15.949476 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:15.949353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.0.1:49500)
0924 19:39:15.949396 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:15.949442 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:39:15.949604 13166 catalog_manager.cc:4584] TS c5894ebe66ea4fa2b14d50b302a14d46 (127.0.98.1:37571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63bc2ce5e22f4df49fa15a9789a22bfd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37571: connect: Connection refused (error 111)
I20260924 19:39:15.950249 13166 catalog_manager.cc:4841] TS 0c57605df4c74a9ea686d77c53fdb092 (127.0.98.2:39979): tablet 63bc2ce5e22f4df49fa15a9789a22bfd (table test-table [id=2dc9a5292ed645dd8df21b50c3fb5782]) successfully deleted
I20260924 19:39:15.950325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13382
I20260924 19:39:15.955466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13515
I20260924 19:39:15.960790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13147
I20260924 19:39:15.967434 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:39:16.301416 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:39:16.301507 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:39:16.635439 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:39:16.635 UTC [12214] LOG: received smart shutdown request
2026-09-24 19:39:16.639 UTC [12214] LOG: background worker "logical replication launcher" (PID 12223) exited with exit code 1
2026-09-24 19:39:16.639 UTC [12218] LOG: shutting down
2026-09-24 19:39:16.792 UTC [12214] LOG: database system is shut down
2026-09-24T19:39:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45656 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:39:16 dist-test-slave-6kf3 krb5kdc[13717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:39:16 dist-test-slave-6kf3 krb5kdc[13717](info): set up 2 sockets
Sep 24 19:39:16 dist-test-slave-6kf3 krb5kdc[13717](info): commencing operation
krb5kdc: starting...
W20260924 19:39:17.819561 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:39:17 dist-test-slave-6kf3 krb5kdc[13717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:39:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:39:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:39:17.926927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:18.004315 13730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:18.004513 13730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:18.004551 13730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:18.005672 13730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:18.005710 13730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:18.005728 13730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:18.005743 13730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:18.005759 13730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:18.007236 13730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:18.007462 13730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:18.007661 13730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:18.012152 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:18.012167 13736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.012197 13730 server_base.cc:1034] running on GCE node
W20260924 19:39:18.012185 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.012633 13730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:18.012908 13730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:18.014043 13730 hybrid_clock.cc:648] HybridClock initialized: now 1790278758014023 us; error 32 us; skew 500 ppm
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:18.016204 13730 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:18.016623 13730 webserver.cc:460] Webserver started at http://127.0.98.62:33111/ using document root <none> and password file <none>
I20260924 19:39:18.016820 13730 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:18.016866 13730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:18.016985 13730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:18.017995 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "247e82e30de14f978f75e67c9ff3e937"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "1dc93a20146a15c9eb3f466673267d9b"
server_key_iv: "d668fd27d0164bf421a7abd1d6241255"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.018322 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "247e82e30de14f978f75e67c9ff3e937"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "1dc93a20146a15c9eb3f466673267d9b"
server_key_iv: "d668fd27d0164bf421a7abd1d6241255"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.019518 13730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:39:18.020197 13745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:18.020355 13730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:18.020433 13730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "247e82e30de14f978f75e67c9ff3e937"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "1dc93a20146a15c9eb3f466673267d9b"
server_key_iv: "d668fd27d0164bf421a7abd1d6241255"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.020489 13730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:18.030591 13730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:18.032405 13730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:18.032534 13730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:18.036263 13730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43313
I20260924 19:39:18.036293 13797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43313 every 8 connection(s)
I20260924 19:39:18.036598 13730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:18.037365 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:18.038806 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:39:18.041038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13730
I20260924 19:39:18.041141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:39:18.041298 392 external_mini_cluster.cc:1428] Setting key 37e3100a3e403fe3c1156c4c590c57b1
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:18.048163 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.039383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43313 (local address 127.0.0.1:56756)
0924 19:39:18.039517 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.039549 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.039998 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.040242 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.040244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.040467 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.040617 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.040621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.041647 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.041648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.042636 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.042639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.043246 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.043247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.043299 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.043746 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.043753 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.045040 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.046834 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.046840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.046842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.047014 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.047253 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.047253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.047254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.047319 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.047698 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.047701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.047753 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.047818 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.047960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260924 19:39:18.048178 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.039524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.0.98.62:43313)
0924 19:39:18.039642 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.039647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.039677 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:39:18.040132 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.040134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.040149 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.040229 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.040682 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.041561 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.042711 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.043179 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.043280 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.045122 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.045130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.045131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.045137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.046760 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.047085 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.047088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.047089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.047196 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.047370 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.047371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.047371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.047634 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.047721 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.047864 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.048029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":90,"thread_start_us":45,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 19:39:18.048867 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:39:18.049167 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.049074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43636)
0924 19:39:18.049113 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.049150 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:39:18.049305 13749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
I20260924 19:39:18.049633 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.049562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43642)
0924 19:39:18.049598 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.049619 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 19:39:18.049738 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38337: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:39:18.063680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38337
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:18.098942 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } attempt: 1
I20260924 19:39:18.099463 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.099326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43658)
0924 19:39:18.099389 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.099433 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:39:18.099802 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.099719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43668)
0924 19:39:18.099757 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.099781 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:39:18.099915 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38337: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
W20260924 19:39:18.141937 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:18.142138 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:18.142158 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:18.143241 13805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:18.143275 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:18.143286 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:18.143297 13805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:18.143308 13805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:18.144780 13805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:18.144968 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:18.145133 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:18.149518 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.149591 13805 server_base.cc:1034] running on GCE node
W20260924 19:39:18.149618 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:18.149686 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.149987 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:18.150233 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:18.151368 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790278758151358 us; error 24 us; skew 500 ppm
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:18.153748 13805 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:39:18.154129 13805 webserver.cc:460] Webserver started at http://127.0.98.61:41581/ using document root <none> and password file <none>
I20260924 19:39:18.154310 13805 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:18.154354 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:18.154471 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:18.155452 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data/instance:
uuid: "e523440dad7a4e63ae5661653fa23317"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "f679d65a0301f29de8c8788770b32916"
server_key_iv: "c11b6ffaba8c4af279b3abe871448b7f"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.155771 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal/instance:
uuid: "e523440dad7a4e63ae5661653fa23317"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "f679d65a0301f29de8c8788770b32916"
server_key_iv: "c11b6ffaba8c4af279b3abe871448b7f"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.156952 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:18.157570 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:18.157708 13805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:18.157786 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "e523440dad7a4e63ae5661653fa23317"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "f679d65a0301f29de8c8788770b32916"
server_key_iv: "c11b6ffaba8c4af279b3abe871448b7f"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.157848 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:18.168087 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } attempt: 2
I20260924 19:39:18.168453 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.168344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43672)
0924 19:39:18.168395 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.168428 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:39:18.168745 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.168671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43684)
0924 19:39:18.168708 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.168730 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:39:18.168824 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38337: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38337: connect: Connection refused (error 111)
I20260924 19:39:18.169437 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:18.171344 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:18.171445 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:18.174988 13805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38337
I20260924 19:39:18.175017 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38337 every 8 connection(s)
I20260924 19:39:18.175312 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:39:18.176136 13873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:18.177731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13805
I20260924 19:39:18.177826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-1/wal/instance
I20260924 19:39:18.177614 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:39:18.178009 392 external_mini_cluster.cc:1428] Setting key dc53fc70292bd8b7c2e252ad5a99033c
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 19:39:18.186959 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.178366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.0.98.62:43313)
0924 19:39:18.178423 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.178425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.178625 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:39:18.179216 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.179217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.179230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.179277 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.180241 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.180746 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.181864 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.182084 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.182150 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.184478 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.184480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.184480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.184487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.185830 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.186209 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.186210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.186210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.186257 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.186435 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.186435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.186435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.186532 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.186576 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.186826 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.186893 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:18.187108 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.178342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43313 (local address 127.0.0.1:56770)
0924 19:39:18.178495 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.178542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.179100 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.179403 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.179408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.179832 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.180131 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.180137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.180844 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.180846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.181771 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.181774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.182163 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.182164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.182287 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.182838 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.182846 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.184367 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.185905 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.185911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.185913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.186143 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.186302 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.186303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.186303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.186392 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.186584 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.186589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.186717 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.186804 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.186922 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 19:39:18.187618 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:39:18.194773 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.187950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43696)
0924 19:39:18.188000 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.188030 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.188071 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.188284 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.188285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.188421 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.188466 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.188468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.189325 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.189326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.189831 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.189833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.190396 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.190397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.190462 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.190851 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.190855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.191970 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.193849 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.193850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.193851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.194006 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.194225 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.194226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.194226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.194270 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.194564 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.194565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.194599 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.194643 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.194710 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:39:18.194896 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.188009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.61:38337)
0924 19:39:18.188158 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.188162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.188177 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:39:18.188193 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.188195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.188212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.188278 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.188538 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.189250 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.189908 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.190319 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.190399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.192045 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.192055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.192055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.192063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.193767 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.194083 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.194087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.194087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.194161 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.194309 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.194310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.194310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.194482 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.194589 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.194685 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.194782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260924 19:39:18.195453 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:39:18.195749 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.195642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56552)
0924 19:39:18.195694 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.195733 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:39:18.195876 13823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
I20260924 19:39:18.196360 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.196262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56564)
0924 19:39:18.196315 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.196341 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 19:39:18.196458 13873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43301: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:39:18.202889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43301
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:18.245677 13873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } attempt: 1
I20260924 19:39:18.246199 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.246073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56570)
0924 19:39:18.246128 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.246168 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:39:18.246490 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.246416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56572)
0924 19:39:18.246449 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.246472 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:39:18.246587 13873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43301: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
I20260924 19:39:18.259999 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } attempt: 3
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:39:18.267150 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.260293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43700)
0924 19:39:18.260347 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.260359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.260410 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.260562 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.260563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.260704 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.260766 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.260768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.261714 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.261715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.262217 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.262219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.262752 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.262754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.262855 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.263328 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.263333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.264606 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.266340 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.266341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.266341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.266496 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.266679 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.266680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.266680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.266722 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.266942 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.266943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.266977 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.267019 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.267085 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:39:18.267233 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.260368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.98.61:38337)
0924 19:39:18.260423 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.260425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.260461 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:39:18.260490 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.260491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.260502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.260557 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.260828 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.261634 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.262289 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.262647 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.262732 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.264689 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.264691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.264691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.264700 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.266251 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.266557 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.266558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.266558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.266619 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.266770 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.266771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.266771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.266890 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.266956 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.267056 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.267142 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:39:18.267534 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:39:18.267822 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.267736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56574)
0924 19:39:18.267775 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.267800 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:39:18.268139 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.268066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56588)
0924 19:39:18.268104 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.268125 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:39:18.268218 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43301: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
W20260924 19:39:18.282162 13880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:18.282341 13880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:18.282361 13880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:18.283443 13880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:18.283481 13880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:18.283494 13880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:18.283519 13880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:18.283528 13880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:18.284958 13880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43313,127.0.98.61:38337,127.0.98.60:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:18.285137 13880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:18.285291 13880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:18.289587 13886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:18.289593 13888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.289772 13880 server_base.cc:1034] running on GCE node
W20260924 19:39:18.289651 13885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:18.290095 13880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:18.290356 13880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:18.291494 13880 hybrid_clock.cc:648] HybridClock initialized: now 1790278758291480 us; error 25 us; skew 500 ppm
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:18.293799 13880 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:39:18.294217 13880 webserver.cc:460] Webserver started at http://127.0.98.60:33661/ using document root <none> and password file <none>
I20260924 19:39:18.294407 13880 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:18.294452 13880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:18.294580 13880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:18.295531 13880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data/instance:
uuid: "1917d8276c8a438db45b3896fa0ed0b5"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "b9b280609bc9b1e57f008f22d41931fb"
server_key_iv: "23de427aa6489d78b5051050e06e3d2e"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.295858 13880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal/instance:
uuid: "1917d8276c8a438db45b3896fa0ed0b5"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "b9b280609bc9b1e57f008f22d41931fb"
server_key_iv: "23de427aa6489d78b5051050e06e3d2e"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.297017 13880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:18.297624 13895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:18.297770 13880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:18.297847 13880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "1917d8276c8a438db45b3896fa0ed0b5"
format_stamp: "Formatted at 2026-09-24 19:39:18 on dist-test-slave-6kf3"
server_key: "b9b280609bc9b1e57f008f22d41931fb"
server_key_iv: "23de427aa6489d78b5051050e06e3d2e"
server_key_version: "encryptionkey@0"
I20260924 19:39:18.297905 13880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:18.299337 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } attempt: 1
I20260924 19:39:18.299645 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.299547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56594)
0924 19:39:18.299590 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.299618 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:39:18.300005 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.299854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56606)
0924 19:39:18.299914 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.299973 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 19:39:18.300096 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43301: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
I20260924 19:39:18.312914 13880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:18.314517 13880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:18.314646 13880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:18.314729 13873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } attempt: 2
I20260924 19:39:18.315047 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.314950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56608)
0924 19:39:18.314996 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.315023 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:39:18.315354 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.315287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56616)
0924 19:39:18.315319 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.315340 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:39:18.315443 13873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43301: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43301: connect: Connection refused (error 111)
I20260924 19:39:18.318609 13880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43301
I20260924 19:39:18.318596 13947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43301 every 8 connection(s)
I20260924 19:39:18.318971 13880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:39:18.319725 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:18.321189 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:18.326977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 13880
I20260924 19:39:18.327123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278584646480-392-0/minicluster-data/master-2/wal/instance
I20260924 19:39:18.327311 392 external_mini_cluster.cc:1428] Setting key 9398aa4ab1e39bcf552aa508fe331bd1
I20260924 19:39:18.329105 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.321804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.98.62:43313)
0924 19:39:18.321860 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.321861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.322036 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:39:18.322483 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.322484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.322492 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.322541 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.323111 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.323567 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.324511 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.324685 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.324735 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.326530 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.326531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.326531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.326538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.327876 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.328247 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.328248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.328249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.328309 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.328529 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.328529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.328530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.328651 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.328699 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.328983 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.329041 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:39:18.329298 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.321764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43313 (local address 127.0.0.1:56776)
0924 19:39:18.321918 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.321956 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.322412 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.322602 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.322605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.322831 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.323031 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.323035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.323627 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.323627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.324452 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.324455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.324740 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.324740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.324833 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.325207 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.325214 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.326444 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.327958 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.327963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.327965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.328183 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.328388 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.328388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.328388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.328473 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.328711 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.328716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.328851 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.328956 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.329096 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260924 19:39:18.329768 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:39:18.335875 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.327638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.0.98.60:43301)
0924 19:39:18.327768 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.327773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.327810 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:39:18.327859 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.327861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.327875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.327962 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.328544 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.329489 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.330343 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.330760 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.330820 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.333197 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.333204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.333204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.333209 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.334760 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335101 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.335104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.335104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.335159 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335338 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.335338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.335339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.335456 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.335577 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.335707 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.335781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260924 19:39:18.336242 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.329991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38337 (local address 127.0.0.1:43716)
0924 19:39:18.330099 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.330108 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.330153 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.330267 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.330267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.330402 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.330450 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.330452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.331225 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.331226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.331999 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.332001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.332352 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.332352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.332406 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.332760 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.332763 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.333937 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.335272 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.335273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.335444 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.335624 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.335625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.335685 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.335965 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.335966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.336004 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.336067 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.336144 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 19:39:18.336249 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.330055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.98.61:38337)
0924 19:39:18.330098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.330099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.330141 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:39:18.330214 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.330215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.330223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.330263 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.330516 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.331154 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.332069 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.332273 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.332339 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.334001 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.334003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.334003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.334011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.335194 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335505 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.335505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.335506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.335565 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.335737 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.335738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.335738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.335859 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.335921 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.336102 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.336169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:39:18.336781 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:18.342379 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.336967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56638)
0924 19:39:18.337004 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.337013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.337058 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.337165 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.337166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.337261 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.337286 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.337286 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.337858 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.337859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.338327 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.338328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.338645 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.338646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.338689 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.339076 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.339080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.340174 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.341535 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.341536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.341537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.341723 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.341895 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.341895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.341895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.341934 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.342135 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.342135 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.342165 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.342206 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.342287 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:18.342427 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.336991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.60:43301)
0924 19:39:18.337021 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.337023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.337050 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:39:18.337094 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.337095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.337104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.337164 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.337344 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.337795 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.338396 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.338594 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.338642 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.340249 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.340250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.340250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.340257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.341471 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.341784 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.341785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.341785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.341834 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.341968 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.341968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.341969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.342081 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.342143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.342232 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.342329 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":13}
I20260924 19:39:18.343343 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.336589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.62:43313)
0924 19:39:18.336623 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.336624 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.336632 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:39:18.336735 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.336736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.336743 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.336784 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.337308 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.337742 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.338358 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.338548 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.338596 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.340805 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.340806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.340806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.340812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.342184 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.342514 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.342514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.342515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.342557 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.342749 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.342750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.342750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.342861 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.343014 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.343199 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.343266 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:39:18.343887 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: Bootstrap starting.
I20260924 19:39:18.344852 13948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:18.345095 13948 log.cc:824] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:18.345814 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: No bootstrap required, opened a new log
I20260924 19:39:18.347663 13948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:39:18.347776 13948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:18.347797 13948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1917d8276c8a438db45b3896fa0ed0b5, State: Initialized, Role: FOLLOWER
I20260924 19:39:18.347918 13948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.348297 13948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:18.348479 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.348553 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:39:18.349905 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:39:18.349994 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:39:18.350677 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.343874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.98.61:38337)
0924 19:39:18.343907 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.343909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.343919 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:39:18.344020 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.344021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.344029 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.344075 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.344606 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.345049 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.345818 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.346015 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.346069 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.348201 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.348202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.348202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.348209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.349607 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.349934 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.349935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.349936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.349988 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.350182 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.350183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.350183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.350312 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.350483 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.350519 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.350591 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:39:18.350147 13880 master_runner.cc:186] Error getting master registration for 127.0.98.62:43313: OK, instance_id { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" instance_seqno: 1790278758035185 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:39:18.352233 13880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:39:18.353781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:39:18.362339 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.354908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.0.98.61:38337)
0924 19:39:18.354942 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.354944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.355183 (+ 239us) server_negotiation.cc:407] Connection header received
0924 19:39:18.355261 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.355262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.355271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.355296 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.356123 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.356816 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.357834 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.358123 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.358154 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.359771 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.359772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.359772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.359778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.361239 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.361592 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.361593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.361593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.361641 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.361820 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.361821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.361821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.361931 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.361993 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.362189 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.362258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:39:18.362478 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.355148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.98.62:43313)
0924 19:39:18.355247 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.355249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.355256 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:39:18.355275 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.355275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.355283 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.355331 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.356124 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.356823 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.357834 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.358116 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.358153 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.359771 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.359772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.359772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.359778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.361414 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.361746 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.361747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.361747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.361792 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.361965 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.361965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.361966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.362072 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.362149 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.362325 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.362391 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:39:18.363735 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.355255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.60:43301)
0924 19:39:18.355371 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.355373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.355468 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:39:18.355553 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.355554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.355562 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.355589 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.356282 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.356969 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.357933 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.358222 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.358259 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.360381 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.360383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.360383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.360390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.362248 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.362896 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.362897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.362898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.362952 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.363309 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.363310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.363310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.363431 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.363511 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.363631 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.363684 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 19:39:18.375236 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } attempt: 2
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:39:18.381577 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.375461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56654)
0924 19:39:18.375501 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.375512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.375582 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.375691 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.375692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.375805 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.375838 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.375839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.376714 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.376715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.377178 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.377180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.377588 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.377589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.377628 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.377999 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.378002 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.379211 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.380850 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.380851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.380852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.380992 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.381155 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.381156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.381156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.381197 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.381385 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.381386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.381415 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.381457 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.381513 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:18.381632 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.375528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.60:43301)
0924 19:39:18.375573 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.375575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.375583 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:39:18.375624 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.375625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.375633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.375688 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.375887 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.376646 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.377249 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.377525 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.377590 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.379282 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.379283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.379284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.379290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.380770 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.381047 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.381049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.381049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.381102 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.381231 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.381231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.381232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.381337 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.381395 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.381481 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.381548 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:39:18.382774 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: Bootstrap starting.
I20260924 19:39:18.383347 13798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:18.383594 13798 log.cc:824] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:18.384263 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: No bootstrap required, opened a new log
I20260924 19:39:18.385658 13798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.385757 13798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:18.385777 13798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247e82e30de14f978f75e67c9ff3e937, State: Initialized, Role: FOLLOWER
I20260924 19:39:18.385888 13798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.386162 13798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:18.386107 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.387468 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:39:18.387816 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:39:18.387887 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:39:18.388525 13730 master_runner.cc:186] Error getting master registration for 127.0.98.61:38337: OK, instance_id { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" instance_seqno: 1790278758174040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:39:18.389998 13730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:39:18.406586 13873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } attempt: 3
Sep 24 19:39:18 dist-test-slave-6kf3 krb5kdc[13717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:39:18.412976 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.406940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.0.98.60:43301)
0924 19:39:18.406980 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:18.406982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:18.406989 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:39:18.407021 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:18.407022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:18.407033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:18.407084 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:18.407368 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.407832 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.408618 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.408822 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.408880 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.410634 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:18.410636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:18.410636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:18.410643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:18.411926 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.412266 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.412267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.412268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.412323 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:18.412492 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:18.412493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:18.412493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:18.412609 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:18.412668 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:18.412821 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:39:18.412887 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:39:18.412976 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:18.406846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43301 (local address 127.0.0.1:56666)
0924 19:39:18.406907 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:18.406919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:18.406960 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:18.407093 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:18.407095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:18.407250 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:18.407294 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.407296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.407917 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.407918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.408470 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:18.408472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:18.408896 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:18.408897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:18.408967 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:18.409377 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:18.409383 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:18.410541 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:18.412031 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.412032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.412033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.412204 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.412379 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:18.412380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:18.412380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:18.412440 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:18.412666 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:18.412667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:18.412715 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:39:18.412776 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:39:18.412875 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:39:18.414321 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: Bootstrap starting.
I20260924 19:39:18.414935 13873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:18.415218 13873 log.cc:824] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:18.416411 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: No bootstrap required, opened a new log
I20260924 19:39:18.417857 13873 raft_consensus.cc:348] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.417968 13873 raft_consensus.cc:374] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:18.418002 13873 raft_consensus.cc:729] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e523440dad7a4e63ae5661653fa23317, State: Initialized, Role: FOLLOWER
I20260924 19:39:18.418135 13873 consensus_queue.cc:260] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.418406 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.418467 13873 sys_catalog.cc:564] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:18.418556 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:39:18.420234 14000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:39:18.420336 14000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:39:18.422153 13826 tablet.cc:2369] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:39:18.422363 13805 master_runner.cc:263] No leader master found from master e523440dad7a4e63ae5661653fa23317
I20260924 19:39:18.422415 13805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:39:18.533277 13751 tablet.cc:2369] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:39:18.565214 13901 tablet.cc:2369] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:39:18.628724 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:18.628836 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.629055 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 247e82e30de14f978f75e67c9ff3e937 (127.0.98.62:43313), e523440dad7a4e63ae5661653fa23317 (127.0.98.61:38337)
I20260924 19:39:18.629309 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1917d8276c8a438db45b3896fa0ed0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247e82e30de14f978f75e67c9ff3e937" is_pre_election: true
I20260924 19:39:18.629328 13847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1917d8276c8a438db45b3896fa0ed0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e523440dad7a4e63ae5661653fa23317" is_pre_election: true
I20260924 19:39:18.629447 13847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1917d8276c8a438db45b3896fa0ed0b5 in term 0.
I20260924 19:39:18.629449 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1917d8276c8a438db45b3896fa0ed0b5 in term 0.
I20260924 19:39:18.629666 13899 leader_election.cc:302] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1917d8276c8a438db45b3896fa0ed0b5, e523440dad7a4e63ae5661653fa23317; no voters:
I20260924 19:39:18.629861 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:39:18.629938 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:39:18.629966 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:18.630739 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.630884 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [CANDIDATE]: Term 1 election: Requested vote from peers 247e82e30de14f978f75e67c9ff3e937 (127.0.98.62:43313), e523440dad7a4e63ae5661653fa23317 (127.0.98.61:38337)
I20260924 19:39:18.631059 13847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1917d8276c8a438db45b3896fa0ed0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e523440dad7a4e63ae5661653fa23317"
I20260924 19:39:18.631062 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1917d8276c8a438db45b3896fa0ed0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247e82e30de14f978f75e67c9ff3e937"
I20260924 19:39:18.631137 13847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:18.631139 13772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:18.631824 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1917d8276c8a438db45b3896fa0ed0b5 in term 1.
I20260924 19:39:18.631820 13847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1917d8276c8a438db45b3896fa0ed0b5 in term 1.
I20260924 19:39:18.632019 13899 leader_election.cc:302] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1917d8276c8a438db45b3896fa0ed0b5, e523440dad7a4e63ae5661653fa23317; no voters:
I20260924 19:39:18.632119 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:18.632282 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [term 1 LEADER]: Becoming Leader. State: Replica: 1917d8276c8a438db45b3896fa0ed0b5, State: Running, Role: LEADER
I20260924 19:39:18.632364 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } }
I20260924 19:39:18.632771 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1917d8276c8a438db45b3896fa0ed0b5. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.632855 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:18.633041 14004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:18.633221 14004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:18.634371 13772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [term 1 FOLLOWER]: Refusing update from remote peer 1917d8276c8a438db45b3896fa0ed0b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:18.634379 13847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [term 1 FOLLOWER]: Refusing update from remote peer 1917d8276c8a438db45b3896fa0ed0b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:18.634634 13954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:18.634727 14002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:18.635775 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1917d8276c8a438db45b3896fa0ed0b5. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.635782 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1917d8276c8a438db45b3896fa0ed0b5. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.635855 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:39:18.635857 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:39:18.636708 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.636782 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:18.636825 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.636862 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:18.636899 14005 mvcc.cc:205] Tried to move back new op lower bound from 7332981795364605952 to 7332981795358728192. Current Snapshot: MvccSnapshot[applied={T|T < 7332981795364605952}]
I20260924 19:39:18.636896 14006 mvcc.cc:205] Tried to move back new op lower bound from 7332981795364605952 to 7332981795358728192. Current Snapshot: MvccSnapshot[applied={T|T < 7332981795364605952}]
I20260924 19:39:18.636883 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.636960 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:39:18.636968 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1917d8276c8a438db45b3896fa0ed0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247e82e30de14f978f75e67c9ff3e937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43313 } } peers { permanent_uuid: "e523440dad7a4e63ae5661653fa23317" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38337 } } peers { permanent_uuid: "1917d8276c8a438db45b3896fa0ed0b5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43301 } } }
I20260924 19:39:18.637034 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:39:18.637079 14007 mvcc.cc:205] Tried to move back new op lower bound from 7332981795364605952 to 7332981795358728192. Current Snapshot: MvccSnapshot[applied={T|T < 7332981795364605952}]
I20260924 19:39:18.637439 14004 catalog_manager.cc:1347] Generated new cluster ID: 4da5d5f9cabd49b6b437a9f16a7b237b
I20260924 19:39:18.637480 14004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:18.655120 14004 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:39:18.655658 14004 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:18.665642 14004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1917d8276c8a438db45b3896fa0ed0b5: Generated new TSK 0
I20260924 19:39:18.665853 14004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:39:19.388330 13986 catalog_manager.cc:1259] Loaded cluster ID: 4da5d5f9cabd49b6b437a9f16a7b237b
I20260924 19:39:19.388401 13986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: loading cluster ID for follower catalog manager: success
I20260924 19:39:19.389169 13986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: acquiring CA information for follower catalog manager: success
I20260924 19:39:19.389418 13986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 247e82e30de14f978f75e67c9ff3e937: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:39:19.420817 14000 catalog_manager.cc:1259] Loaded cluster ID: 4da5d5f9cabd49b6b437a9f16a7b237b
I20260924 19:39:19.420892 14000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: loading cluster ID for follower catalog manager: success
I20260924 19:39:19.421674 14000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: acquiring CA information for follower catalog manager: success
I20260924 19:39:19.421931 14000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e523440dad7a4e63ae5661653fa23317: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:39:19.731792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13730
I20260924 19:39:19.736145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13805
I20260924 19:39:19.740128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 13880
2026-09-24T19:39:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2865 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:39:19 dist-test-slave-6kf3 krb5kdc[14019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:39:19 dist-test-slave-6kf3 krb5kdc[14019](info): set up 2 sockets
Sep 24 19:39:19 dist-test-slave-6kf3 krb5kdc[14019](info): commencing operation
krb5kdc: starting...
W20260924 19:39:20.699672 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:39:20 dist-test-slave-6kf3 krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:39:20.804211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:20.881691 14032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:20.881880 14032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:20.881901 14032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:20.883002 14032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:20.883042 14032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:20.883060 14032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:20.883078 14032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:20.883088 14032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:20.884563 14032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:20.884790 14032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:20.884991 14032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:20.889372 14037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:20.889400 14038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:20.889528 14032 server_base.cc:1034] running on GCE node
W20260924 19:39:20.889393 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:20.889775 14032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:20.890019 14032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:20.891181 14032 hybrid_clock.cc:648] HybridClock initialized: now 1790278760891167 us; error 27 us; skew 500 ppm
Sep 24 19:39:20 dist-test-slave-6kf3 krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:20.893428 14032 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:20.893811 14032 webserver.cc:460] Webserver started at http://127.0.98.62:44777/ using document root <none> and password file <none>
I20260924 19:39:20.894004 14032 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:20.894049 14032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:20.894165 14032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:20.895108 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "788284f537294af5865eb67fade423a6"
format_stamp: "Formatted at 2026-09-24 19:39:20 on dist-test-slave-6kf3"
server_key: "ac51098112599ec3eaffd7886151f28b"
server_key_iv: "b42f87f84db961f9366dc6f90d8d47ac"
server_key_version: "encryptionkey@0"
I20260924 19:39:20.895414 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "788284f537294af5865eb67fade423a6"
format_stamp: "Formatted at 2026-09-24 19:39:20 on dist-test-slave-6kf3"
server_key: "ac51098112599ec3eaffd7886151f28b"
server_key_iv: "b42f87f84db961f9366dc6f90d8d47ac"
server_key_version: "encryptionkey@0"
I20260924 19:39:20.896678 14032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:39:20.897342 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:20.897505 14032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:20.897579 14032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "788284f537294af5865eb67fade423a6"
format_stamp: "Formatted at 2026-09-24 19:39:20 on dist-test-slave-6kf3"
server_key: "ac51098112599ec3eaffd7886151f28b"
server_key_iv: "b42f87f84db961f9366dc6f90d8d47ac"
server_key_version: "encryptionkey@0"
I20260924 19:39:20.897634 14032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:20.916783 14032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:20.918545 14032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:20.918669 14032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:20.922256 14032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44363
I20260924 19:39:20.922302 14099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44363 every 8 connection(s)
I20260924 19:39:20.922605 14032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:20.923417 14100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:20.925740 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6: Bootstrap starting.
I20260924 19:39:20.926246 14100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:20.926473 14100 log.cc:824] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:20.927086 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6: No bootstrap required, opened a new log
I20260924 19:39:20.928299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14032
I20260924 19:39:20.928392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:39:20.928551 392 external_mini_cluster.cc:1428] Setting key 867b23ab3873b4e9c0d5fda24b7bd8a1
I20260924 19:39:20.928802 14100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260924 19:39:20.928979 14100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:20.929071 14100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788284f537294af5865eb67fade423a6, State: Initialized, Role: FOLLOWER
I20260924 19:39:20.929209 14100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260924 19:39:20.929278 14100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:20.929301 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:20.929350 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:20.930428 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260924 19:39:20.930527 14100 leader_election.cc:302] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788284f537294af5865eb67fade423a6; no voters:
I20260924 19:39:20.930694 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:39:20.930737 14105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:20.930855 14105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [term 1 LEADER]: Becoming Leader. State: Replica: 788284f537294af5865eb67fade423a6, State: Running, Role: LEADER
I20260924 19:39:20.930958 14100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:20.931008 14105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260924 19:39:20.931432 14105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788284f537294af5865eb67fade423a6. Latest consensus state: current_term: 1 leader_uuid: "788284f537294af5865eb67fade423a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260924 19:39:20.931439 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788284f537294af5865eb67fade423a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788284f537294af5865eb67fade423a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260924 19:39:20.931596 14105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:20.931604 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:20.931921 14114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:20.932770 14114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:20.934322 14114 catalog_manager.cc:1347] Generated new cluster ID: e94f17c50ef64bc5a0538488e3075258
I20260924 19:39:20.934377 14114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:39:20 dist-test-slave-6kf3 krb5kdc[14019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:20.939694 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:20.928980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.98.62:44363)
0924 19:39:20.929156 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:20.929165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:20.929205 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:39:20.929319 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:20.929326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:20.929378 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:20.929499 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:20.930281 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:20.931429 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:20.932588 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:20.933147 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:20.933274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:20.935922 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:20.935944 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:20.935945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:20.935956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:20.937851 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:20.938256 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:20.938260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:20.938260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:20.938417 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:20.938625 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:20.938626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:20.938626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:20.938928 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:20.939088 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:20.939186 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:39:20.939465 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260924 19:39:20.939698 14114 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:39:20.940359 14114 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:20.940668 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:39:20.955595 14114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788284f537294af5865eb67fade423a6: Generated new TSK 0
I20260924 19:39:20.955782 14114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:39:20.994457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14032
2026-09-24T19:39:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1253 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:39:21 dist-test-slave-6kf3 krb5kdc[14127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:39:21 dist-test-slave-6kf3 krb5kdc[14127](info): set up 2 sockets
Sep 24 19:39:21 dist-test-slave-6kf3 krb5kdc[14127](info): commencing operation
krb5kdc: starting...
W20260924 19:39:21.931596 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:39:22.034647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:22.111192 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:22.111384 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:22.111406 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:22.112545 14140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:22.112583 14140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:22.112596 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:22.112607 14140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:22.112618 14140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:22.113986 14140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:22.114168 14140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:22.114343 14140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:22.118826 14148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.118841 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.118821 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.119025 14140 server_base.cc:1034] running on GCE node
I20260924 19:39:22.119261 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:22.119525 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:22.120699 14140 hybrid_clock.cc:648] HybridClock initialized: now 1790278762120685 us; error 24 us; skew 500 ppm
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:22.122891 14140 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:22.123270 14140 webserver.cc:460] Webserver started at http://127.0.98.62:43017/ using document root <none> and password file <none>
I20260924 19:39:22.123464 14140 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:22.123508 14140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:22.123669 14140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:22.124693 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "602e3421ca8e42da80818268c95a7f6e"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "8ff4a6f45ef25e55a0169d6323458c15"
server_key_iv: "4da113e7a25634ac57061b3cdee15157"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.125003 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "602e3421ca8e42da80818268c95a7f6e"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "8ff4a6f45ef25e55a0169d6323458c15"
server_key_iv: "4da113e7a25634ac57061b3cdee15157"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.126178 14140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:39:22.126888 14155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.127031 14140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:22.127110 14140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "602e3421ca8e42da80818268c95a7f6e"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "8ff4a6f45ef25e55a0169d6323458c15"
server_key_iv: "4da113e7a25634ac57061b3cdee15157"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.127172 14140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:22.141407 14140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:22.143129 14140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:22.143267 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:22.147234 14140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44217
I20260924 19:39:22.147269 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44217 every 8 connection(s)
I20260924 19:39:22.147634 14140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:22.148589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14140
I20260924 19:39:22.148651 14208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:22.148715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:39:22.148953 392 external_mini_cluster.cc:1428] Setting key a5de8cde74d8747f8a3cb749096fa63f
I20260924 19:39:22.151861 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap starting.
I20260924 19:39:22.152591 14208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:22.152894 14208 log.cc:824] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:22.153612 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: No bootstrap required, opened a new log
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:22.156173 14208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.156348 14208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:22.156405 14208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Initialized, Role: FOLLOWER
I20260924 19:39:22.156541 14208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.156610 14208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:22.156634 14208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:22.156685 14208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:22.157256 14208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.157354 14208 leader_election.cc:302] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602e3421ca8e42da80818268c95a7f6e; no voters:
I20260924 19:39:22.157502 14208 leader_election.cc:288] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:39:22.157546 14213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:22.157714 14213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 LEADER]: Becoming Leader. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Running, Role: LEADER
I20260924 19:39:22.157768 14208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:22.157886 14213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.158710 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:22.158972 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602e3421ca8e42da80818268c95a7f6e. Latest consensus state: current_term: 1 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:22.159174 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:22.159149 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:22.159600 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.149463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.98.62:44217)
0924 19:39:22.149604 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.149612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.149636 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:39:22.150154 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.150161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.150224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.150330 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:22.151109 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.152021 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.153006 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.153399 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.153480 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.155993 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:22.155997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:22.155997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:22.156006 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:22.157868 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.158265 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.158267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.158268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.158395 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.158631 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.158631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.158632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.158933 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:22.159078 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.159261 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.159399 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
W20260924 19:39:22.159799 14228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:39:22.159847 14228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:39:22.159973 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:22.160148 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:22.161473 14229 catalog_manager.cc:1347] Generated new cluster ID: 6c8e1fdc768c48719a0f13baf55145a1
I20260924 19:39:22.161511 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:22.178081 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:39:22.178570 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:22.185060 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Generated new TSK 0
I20260924 19:39:22.185216 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:39:22.217947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:22.313019 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:22.313215 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:22.313238 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:22.314355 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:22.314394 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:22.314422 14234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:22.315848 14234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:22.316092 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:22.316293 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:22.320932 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.320955 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.321039 14234 server_base.cc:1034] running on GCE node
W20260924 19:39:22.321130 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.321312 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:22.321496 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:22.322631 14234 hybrid_clock.cc:648] HybridClock initialized: now 1790278762322617 us; error 25 us; skew 500 ppm
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:22.324908 14234 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:22.325296 14234 webserver.cc:460] Webserver started at http://127.0.98.1:38915/ using document root <none> and password file <none>
I20260924 19:39:22.325493 14234 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:22.325538 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:22.325657 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:22.326633 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a89114615bd4372ab6281a3554fa6d9"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "af78354251c318260c1d8b0741929c56"
server_key_iv: "801721ccb20b27cf5bcc0bb19229da57"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.326946 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a89114615bd4372ab6281a3554fa6d9"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "af78354251c318260c1d8b0741929c56"
server_key_iv: "801721ccb20b27cf5bcc0bb19229da57"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.328105 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:39:22.328737 14249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.328879 14234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.328955 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "9a89114615bd4372ab6281a3554fa6d9"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "af78354251c318260c1d8b0741929c56"
server_key_iv: "801721ccb20b27cf5bcc0bb19229da57"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.329011 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:22.344462 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:22.346441 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:22.346570 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:22.346765 14234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:22.347038 14234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:22.347071 14234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.347110 14234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:22.347133 14234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.352286 14234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44359
I20260924 19:39:22.352342 14362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44359 every 8 connection(s)
I20260924 19:39:22.352602 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:22.360970 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.353259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45719 (local address 127.0.98.62:44217)
0924 19:39:22.353313 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.353315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.353512 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:39:22.353983 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.353985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.353998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.354047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:22.354922 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.355413 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.356384 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.356590 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.356645 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.358517 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:22.358519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:22.358519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:22.358525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:22.359763 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.360131 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.360132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.360132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.360179 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.360356 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.360356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.360356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.360470 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:22.360527 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.360801 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.360881 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:22.361138 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.353247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:45719)
0924 19:39:22.353396 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.353430 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.353891 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.354111 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.354114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.354637 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:22.354826 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.354830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.355492 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.355493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.356310 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.356312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.356656 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.356657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.356766 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.357139 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:22.357147 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:22.358447 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:22.359829 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.359835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.359837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.360073 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.360228 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.360229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.360229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.360320 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.360530 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:22.360554 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:22.360682 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.360770 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.360936 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 19:39:22.361573 14363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:22.361651 14363 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:22.361783 14363 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:22.362213 14172 ts_manager.cc:194] Registered new tserver with Master: 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359)
I20260924 19:39:22.362694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14234
I20260924 19:39:22.362780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:39:22.362787 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45719
I20260924 19:39:22.362973 392 external_mini_cluster.cc:1428] Setting key 85521f687be9320c2637a12d6bb8b67c
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:39:22.383663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:22.460395 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:22.460577 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:22.460598 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:22.461726 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:22.461767 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:22.461791 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:22.463129 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:22.463294 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:22.463454 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:22.467955 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.467902 14376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.467983 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.468043 14368 server_base.cc:1034] running on GCE node
I20260924 19:39:22.468259 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:22.468473 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:22.469604 14368 hybrid_clock.cc:648] HybridClock initialized: now 1790278762469588 us; error 27 us; skew 500 ppm
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:22.471915 14368 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:22.472339 14368 webserver.cc:460] Webserver started at http://127.0.98.2:45929/ using document root <none> and password file <none>
I20260924 19:39:22.472500 14368 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:22.472536 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:22.472621 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:22.473533 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b93a0bd72564816af9efc67efc08959"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "00605bbec0bb3e1b89c8f5825fe8b151"
server_key_iv: "a77ac739021199e065eee7cb8623cac2"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.473812 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b93a0bd72564816af9efc67efc08959"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "00605bbec0bb3e1b89c8f5825fe8b151"
server_key_iv: "a77ac739021199e065eee7cb8623cac2"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.474877 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:39:22.475452 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.475589 14368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:22.475663 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "5b93a0bd72564816af9efc67efc08959"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "00605bbec0bb3e1b89c8f5825fe8b151"
server_key_iv: "a77ac739021199e065eee7cb8623cac2"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.475728 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:22.492218 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:22.494150 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:22.494272 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:22.494449 14368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:22.494724 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:22.494755 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.494781 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:22.494804 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.499957 14368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42869
I20260924 19:39:22.500006 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42869 every 8 connection(s)
I20260924 19:39:22.500279 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:22.507620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14368
I20260924 19:39:22.507700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:39:22.507874 392 external_mini_cluster.cc:1428] Setting key 2a4a7194ea911431a3e2dfa875c29b7b
I20260924 19:39:22.508440 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.500886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60003 (local address 127.0.98.62:44217)
0924 19:39:22.500941 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.500944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.501128 (+ 184us) server_negotiation.cc:407] Connection header received
0924 19:39:22.501581 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.501583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.501594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.501651 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:22.502509 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.502994 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.503999 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.504193 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.504237 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.506075 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:22.506076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:22.506077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:22.506083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:22.507119 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.507460 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.507461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.507462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.507512 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.507804 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.507805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.507805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.507912 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:22.508009 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.508275 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.508353 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:22.508677 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.500854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:60003)
0924 19:39:22.501010 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.501050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.501494 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.501712 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.501715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.502245 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:22.502424 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.502428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.503069 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.503070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.503893 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.503896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.504246 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.504246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.504339 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.504750 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:22.504759 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:22.505995 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:22.507178 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.507183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.507185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.507402 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.507610 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.507611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.507611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.507728 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.508002 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:22.508008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:22.508154 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.508256 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.508409 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 19:39:22.509078 14497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:22.509156 14497 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:22.509308 14497 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:22.509642 14172 ts_manager.cc:194] Registered new tserver with Master: 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869)
I20260924 19:39:22.510037 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60003
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:39:22.528453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:22.605785 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:22.605983 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:22.606012 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:22.607110 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:22.607147 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:22.607182 14502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:22.608594 14502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:22.608824 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:22.609016 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:22.613567 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.613583 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.613755 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.613677 14502 server_base.cc:1034] running on GCE node
I20260924 19:39:22.613993 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:22.614221 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:22.615357 14502 hybrid_clock.cc:648] HybridClock initialized: now 1790278762615348 us; error 24 us; skew 500 ppm
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:22.617667 14502 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:22.618057 14502 webserver.cc:460] Webserver started at http://127.0.98.3:39701/ using document root <none> and password file <none>
I20260924 19:39:22.618257 14502 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:22.618310 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:22.618422 14502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:22.619369 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "e3a79941bd0061b3a2f01460ce3a7946"
server_key_iv: "5b3f3b6e178d5755fe69fcf98098e59d"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.619670 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "e3a79941bd0061b3a2f01460ce3a7946"
server_key_iv: "5b3f3b6e178d5755fe69fcf98098e59d"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.620854 14502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:39:22.621491 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.621631 14502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:22.621703 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "e3a79941bd0061b3a2f01460ce3a7946"
server_key_iv: "5b3f3b6e178d5755fe69fcf98098e59d"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.621765 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:22.635668 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:22.637225 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:22.637341 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:22.637519 14502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:22.637777 14502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:22.637807 14502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.637827 14502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:22.637850 14502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.643086 14502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35243
I20260924 19:39:22.643139 14630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35243 every 8 connection(s)
I20260924 19:39:22.643402 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:22.652375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14502
I20260924 19:39:22.652478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:39:22.652674 392 external_mini_cluster.cc:1428] Setting key c98db36b972a4b9988da3e4ae410536c
I20260924 19:39:22.656762 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.644005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50671 (local address 127.0.98.62:44217)
0924 19:39:22.644073 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.644075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.644274 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:39:22.644701 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.644702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.644711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.644757 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:22.645628 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.646105 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.647054 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.647244 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.647296 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.649222 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:22.649224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:22.649224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:22.649232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:22.655620 (+ 6388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.656000 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.656000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.656001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.656055 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.656275 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.656275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.656275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.656384 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:22.656426 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.656640 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.656699 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:39:22.656941 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.643997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:50671)
0924 19:39:22.644157 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.644200 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.644628 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.644808 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.644811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.645369 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:22.645541 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.645545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.646179 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.646180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.646978 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.646980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.647305 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.647306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.647425 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.647842 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:22.647850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:22.649127 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:22.655690 (+ 6563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.655697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.655699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.655916 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.656102 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:22.656102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:22.656103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:22.656220 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:22.656424 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:22.656427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:22.656542 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.656614 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.656749 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 19:39:22.657335 14631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:22.657445 14631 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:22.657616 14631 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:22.657984 14172 ts_manager.cc:194] Registered new tserver with Master: 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
I20260924 19:39:22.658420 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50671
I20260924 19:39:22.664816 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:39:22.671290 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.666001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.62:44217)
0924 19:39:22.666057 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.666058 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.666141 (+ 83us) server_negotiation.cc:407] Connection header received
0924 19:39:22.666207 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.666207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.666216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.666259 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:22.666754 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.667224 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.667900 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.668139 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.668198 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.669248 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:22.669249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:22.669250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:22.669256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:22.670317 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.670593 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.670594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.670594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.670645 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:22.670803 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:22.670804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:22.670805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:22.670940 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:22.671023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.671126 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.671205 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:39:22.672784 14172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:39:22.674270 14172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:39:22.680311 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.98.1:44359)
0924 19:39:22.676478 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.676483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.676581 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:39:22.676670 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.676672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.676687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.676759 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:22.676949 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.678082 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.679314 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.679953 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680010 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680069 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.680185 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.680226 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260924 19:39:22.680346 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.0.1:41736)
0924 19:39:22.676683 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.676690 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.676717 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.676966 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.676967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.676969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:22.677019 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.677020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.678290 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.678291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.679086 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.679088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.680050 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.680123 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680126 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.680155 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.680248 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":37,"thread_start_us":35,"threads_started":1}
I20260924 19:39:22.680337 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44359 (local address 127.0.0.1:51626)
0924 19:39:22.676477 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.676515 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.676581 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.676786 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.676789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.676791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:22.676861 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.676862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.678179 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.678179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.679235 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.679237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.680009 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.680088 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680102 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.680165 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.680243 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":62,"threads_started":1}
I20260924 19:39:22.680434 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.0.1:33672)
0924 19:39:22.676478 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.676516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.676577 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.676871 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.676872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.676874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:22.676918 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.676920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.677785 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.677786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.679064 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.679067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.680052 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.680121 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680125 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.680155 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.680378 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":187,"thread_start_us":63,"threads_started":1}
I20260924 19:39:22.680560 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.2:42869)
0924 19:39:22.676732 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.676736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.676745 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:22.676780 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.676782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.676794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.676915 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:22.676983 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.677703 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.679150 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.679954 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680197 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680279 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.680305 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.680346 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":82,"threads_started":1}
I20260924 19:39:22.680801 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.676482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.0.98.3:35243)
0924 19:39:22.676817 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.676821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.676830 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:22.676862 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.676863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.676879 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.677083 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:22.677147 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.678202 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.679175 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.679953 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.680575 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.680643 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.680673 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.680726 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":88,"threads_started":1}
I20260924 19:39:22.681253 14431 tablet_service.cc:1467] Processing CreateTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 (DEFAULT_TABLE table=test-table [id=66c24a92526649d18401cbbe0d149243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:22.681373 14297 tablet_service.cc:1467] Processing CreateTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 (DEFAULT_TABLE table=test-table [id=66c24a92526649d18401cbbe0d149243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:22.681465 14565 tablet_service.cc:1467] Processing CreateTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 (DEFAULT_TABLE table=test-table [id=66c24a92526649d18401cbbe0d149243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:22.681577 14431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ce26148c7846a1b6e9e43b428273b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:22.681697 14297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ce26148c7846a1b6e9e43b428273b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:22.681712 14565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ce26148c7846a1b6e9e43b428273b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:22.684165 14645 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Bootstrap starting.
I20260924 19:39:22.684769 14646 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Bootstrap starting.
I20260924 19:39:22.684978 14645 tablet_bootstrap.cc:654] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:22.685461 14646 tablet_bootstrap.cc:654] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:22.685580 14645 log.cc:824] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:22.685959 14646 log.cc:824] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:22.686266 14647 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap starting.
I20260924 19:39:22.686336 14645 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: No bootstrap required, opened a new log
I20260924 19:39:22.686398 14645 ts_tablet_manager.cc:1397] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:22.686739 14646 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: No bootstrap required, opened a new log
I20260924 19:39:22.686805 14646 ts_tablet_manager.cc:1397] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:22.686892 14647 tablet_bootstrap.cc:654] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:22.687395 14647 log.cc:824] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:22.688177 14647 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: No bootstrap required, opened a new log
I20260924 19:39:22.688303 14647 ts_tablet_manager.cc:1397] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:22.688341 14645 raft_consensus.cc:348] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.688517 14645 raft_consensus.cc:374] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:22.688551 14645 raft_consensus.cc:729] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93a0bd72564816af9efc67efc08959, State: Initialized, Role: FOLLOWER
I20260924 19:39:22.688576 14646 raft_consensus.cc:348] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.688712 14645 consensus_queue.cc:260] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.688748 14646 raft_consensus.cc:374] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:22.688791 14646 raft_consensus.cc:729] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, State: Initialized, Role: FOLLOWER
I20260924 19:39:22.689000 14645 ts_tablet_manager.cc:1428] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:39:22.688957 14646 consensus_queue.cc:260] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.689026 14497 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
I20260924 19:39:22.689226 14646 ts_tablet_manager.cc:1428] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:39:22.689273 14631 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
I20260924 19:39:22.690105 14647 raft_consensus.cc:348] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.690414 14647 raft_consensus.cc:374] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:22.690460 14647 raft_consensus.cc:729] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a89114615bd4372ab6281a3554fa6d9, State: Initialized, Role: FOLLOWER
I20260924 19:39:22.690551 14647 consensus_queue.cc:260] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.690778 14647 ts_tablet_manager.cc:1428] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:22.690819 14363 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
W20260924 19:39:22.750804 14498 tablet.cc:2369] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:39:22.770089 14653 raft_consensus.cc:482] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:22.770234 14653 raft_consensus.cc:504] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.770593 14653 leader_election.cc:288] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
I20260924 19:39:22.774230 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.770867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58785 (local address 127.0.98.3:35243)
0924 19:39:22.770908 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.770910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.771059 (+ 149us) server_negotiation.cc:407] Connection header received
0924 19:39:22.771093 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.771094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.771105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.771165 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:22.771281 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.772045 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773297 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.773904 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773970 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.773978 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.774115 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.774156 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:39:22.774230 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.770831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33337 (local address 127.0.98.2:42869)
0924 19:39:22.770881 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.770883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.770893 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:39:22.770922 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.770923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.770935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.771004 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:22.771164 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.771910 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773298 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.773868 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773953 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.773965 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.774115 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.774157 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:39:22.774232 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.770742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.98.1:33337)
0924 19:39:22.770787 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.770800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.770843 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.771021 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.771022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.771024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:22.771097 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.771099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.772015 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.772016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.773194 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.773197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.773947 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.774045 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.774050 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.774100 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.774154 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":19}
I20260924 19:39:22.774247 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.770806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.98.1:58785)
0924 19:39:22.770981 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.770989 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:22.771037 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:22.771171 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:22.771172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:22.771173 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:22.771226 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.771227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.772110 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.772110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.773203 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:22.773205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.773973 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.773974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:22.774045 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.774050 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:39:22.774100 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:39:22.774149 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260924 19:39:22.774603 14451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ce26148c7846a1b6e9e43b428273b2" candidate_uuid: "9a89114615bd4372ab6281a3554fa6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93a0bd72564816af9efc67efc08959" is_pre_election: true
I20260924 19:39:22.774626 14585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ce26148c7846a1b6e9e43b428273b2" candidate_uuid: "9a89114615bd4372ab6281a3554fa6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" is_pre_election: true
I20260924 19:39:22.774744 14585 raft_consensus.cc:2454] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a89114615bd4372ab6281a3554fa6d9 in term 0.
I20260924 19:39:22.774751 14451 raft_consensus.cc:2454] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a89114615bd4372ab6281a3554fa6d9 in term 0.
I20260924 19:39:22.774958 14250 leader_election.cc:302] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93a0bd72564816af9efc67efc08959, 9a89114615bd4372ab6281a3554fa6d9; no voters:
I20260924 19:39:22.775092 14653 raft_consensus.cc:2781] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:39:22.775158 14653 raft_consensus.cc:482] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:39:22.775185 14653 raft_consensus.cc:3037] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:22.775887 14653 raft_consensus.cc:504] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.776054 14653 leader_election.cc:288] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 1 election: Requested vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
I20260924 19:39:22.776177 14451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ce26148c7846a1b6e9e43b428273b2" candidate_uuid: "9a89114615bd4372ab6281a3554fa6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b93a0bd72564816af9efc67efc08959"
I20260924 19:39:22.776211 14585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ce26148c7846a1b6e9e43b428273b2" candidate_uuid: "9a89114615bd4372ab6281a3554fa6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
I20260924 19:39:22.776259 14451 raft_consensus.cc:3037] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:22.776269 14585 raft_consensus.cc:3037] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:22.776902 14585 raft_consensus.cc:2454] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a89114615bd4372ab6281a3554fa6d9 in term 1.
I20260924 19:39:22.776934 14451 raft_consensus.cc:2454] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a89114615bd4372ab6281a3554fa6d9 in term 1.
I20260924 19:39:22.777072 14253 leader_election.cc:302] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 9a89114615bd4372ab6281a3554fa6d9; no voters:
I20260924 19:39:22.777194 14653 raft_consensus.cc:2781] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:22.777364 14653 raft_consensus.cc:686] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 9a89114615bd4372ab6281a3554fa6d9, State: Running, Role: LEADER
I20260924 19:39:22.777456 14653 consensus_queue.cc:237] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:22.778258 14171 catalog_manager.cc:5495] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9a89114615bd4372ab6281a3554fa6d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } health_report { overall_health: UNKNOWN } } }
I20260924 19:39:22.794823 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.792748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.98.1:44359)
0924 19:39:22.792794 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:22.792796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:22.792803 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:39:22.792867 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:22.792868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:22.792884 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:22.792936 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:39:22.793049 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.793476 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.794212 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:22.794378 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:22.794415 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:22.794587 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:39:22.794671 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:39:22.794727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:22.794755 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 19:39:22.794770 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:39:22.796407 14585 raft_consensus.cc:1261] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Refusing update from remote peer 9a89114615bd4372ab6281a3554fa6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:22.796409 14451 raft_consensus.cc:1261] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Refusing update from remote peer 9a89114615bd4372ab6281a3554fa6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:22.796679 14653 consensus_queue.cc:1035] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:22.796815 14655 consensus_queue.cc:1035] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:22.798945 14658 mvcc.cc:205] Tried to move back new op lower bound from 7332981812411650048 to 7332981812337008640. Current Snapshot: MvccSnapshot[applied={T|T < 7332981812411650048}]
I20260924 19:39:22.799340 14659 mvcc.cc:205] Tried to move back new op lower bound from 7332981812411650048 to 7332981812337008640. Current Snapshot: MvccSnapshot[applied={T|T < 7332981812411650048 or (T in {7332981812411650048})}]
I20260924 19:39:22.801581 14172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:22.801689 14172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:22.802385 14172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Sending DeleteTablet for 3 replicas of tablet d3ce26148c7846a1b6e9e43b428273b2
I20260924 19:39:22.802670 14431 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33672
I20260924 19:39:22.802670 14297 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51626
I20260924 19:39:22.802748 14565 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41736
I20260924 19:39:22.802958 14668 tablet_replica.cc:330] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: stopping tablet replica
I20260924 19:39:22.803005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14234
I20260924 19:39:22.803056 14668 raft_consensus.cc:2229] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:39:22.803139 14668 raft_consensus.cc:2258] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:22.803664 14667 tablet_replica.cc:330] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: stopping tablet replica
I20260924 19:39:22.803753 14667 raft_consensus.cc:2229] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:22.803838 14667 raft_consensus.cc:2258] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:22.804390 14669 tablet_replica.cc:330] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: stopping tablet replica
I20260924 19:39:22.804474 14669 raft_consensus.cc:2229] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:22.804556 14669 raft_consensus.cc:2258] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:22.804633 14667 ts_tablet_manager.cc:1905] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:22.804888 14669 ts_tablet_manager.cc:1905] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:22.806231 14669 ts_tablet_manager.cc:1918] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:22.806299 14669 log.cc:1196] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/d3ce26148c7846a1b6e9e43b428273b2
I20260924 19:39:22.806284 14667 ts_tablet_manager.cc:1918] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:22.806345 14667 log.cc:1196] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/d3ce26148c7846a1b6e9e43b428273b2
I20260924 19:39:22.806548 14669 ts_tablet_manager.cc:1939] T d3ce26148c7846a1b6e9e43b428273b2 P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting consensus metadata
I20260924 19:39:22.806622 14667 ts_tablet_manager.cc:1939] T d3ce26148c7846a1b6e9e43b428273b2 P 5b93a0bd72564816af9efc67efc08959: Deleting consensus metadata
I20260924 19:39:22.806914 14159 catalog_manager.cc:4841] TS 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): tablet d3ce26148c7846a1b6e9e43b428273b2 (table test-table [id=66c24a92526649d18401cbbe0d149243]) successfully deleted
I20260924 19:39:22.807018 14156 catalog_manager.cc:4841] TS 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): tablet d3ce26148c7846a1b6e9e43b428273b2 (table test-table [id=66c24a92526649d18401cbbe0d149243]) successfully deleted
W20260924 19:39:22.808383 14159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44359 (error 108)
I20260924 19:39:22.808488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14368
I20260924 19:39:22.809336 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:22.809230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44359 (local address 127.0.0.1:51638)
0924 19:39:22.809275 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:22.809310 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:39:22.809458 14159 catalog_manager.cc:4584] TS 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3ce26148c7846a1b6e9e43b428273b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44359: connect: Connection refused (error 111)
I20260924 19:39:22.813136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14502
I20260924 19:39:22.817277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14140
I20260924 19:39:22.821779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--webserver_interface=127.0.98.62
--webserver_port=43017
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:22.899840 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:22.900066 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:22.900092 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:22.901192 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:22.901227 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:22.901239 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:22.901250 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:22.901261 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:22.902616 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:22.902791 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:22.902948 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:22.910912 14678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:22.910934 14677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.911124 14671 server_base.cc:1034] running on GCE node
W20260924 19:39:22.910912 14680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:22.911432 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:22.911713 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:22.912851 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790278762912838 us; error 34 us; skew 500 ppm
Sep 24 19:39:22 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:22.915138 14671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:22.915565 14671 webserver.cc:460] Webserver started at http://127.0.98.62:43017/ using document root <none> and password file <none>
I20260924 19:39:22.915771 14671 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:22.915822 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:22.917107 14671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:22.917678 14687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:22.917812 14671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:22.917886 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "602e3421ca8e42da80818268c95a7f6e"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "8ff4a6f45ef25e55a0169d6323458c15"
server_key_iv: "4da113e7a25634ac57061b3cdee15157"
server_key_version: "encryptionkey@0"
I20260924 19:39:22.918252 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:22.926232 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:22.927843 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:22.927965 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:22.931509 14671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44217
I20260924 19:39:22.931538 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44217 every 8 connection(s)
I20260924 19:39:22.931869 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:22.933388 14740 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:39:22.933997 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap starting.
I20260924 19:39:22.935775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14671
I20260924 19:39:22.936034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44359
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=38915
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:22.942085 14740 log.cc:824] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:22.944159 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:39:22.944430 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap complete.
I20260924 19:39:22.947081 14740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.947331 14740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Initialized, Role: FOLLOWER
I20260924 19:39:22.947489 14740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.947551 14740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:22.947600 14740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:22.947652 14740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:39:22.948704 14740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.948802 14740 leader_election.cc:302] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602e3421ca8e42da80818268c95a7f6e; no voters:
I20260924 19:39:22.948974 14740 leader_election.cc:288] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:39:22.949026 14744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:39:22.949154 14744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 LEADER]: Becoming Leader. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Running, Role: LEADER
I20260924 19:39:22.949259 14740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:22.950230 14744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:22.950736 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:22.950839 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:22.950736 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602e3421ca8e42da80818268c95a7f6e. Latest consensus state: current_term: 2 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:22.951033 14749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:22.951066 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:22.951812 14749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:22.952040 14749 catalog_manager.cc:1259] Loaded cluster ID: 6c8e1fdc768c48719a0f13baf55145a1
I20260924 19:39:22.952257 14749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:22.952471 14749 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:39:22.955080 14749 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:39:22.956801 14749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:22.956982 14749 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Loaded TSK: 0
I20260924 19:39:22.957183 14749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:39:23.026372 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.026571 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.026602 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.027716 14742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.027755 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.027788 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:23.029218 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38915
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.029435 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.029623 14742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.034123 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.034119 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.034271 14770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.034474 14742 server_base.cc:1034] running on GCE node
I20260924 19:39:23.034677 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.034945 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.036105 14742 hybrid_clock.cc:648] HybridClock initialized: now 1790278763036086 us; error 30 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.039058 14742 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:23.039592 14742 webserver.cc:460] Webserver started at http://127.0.98.1:38915/ using document root <none> and password file <none>
I20260924 19:39:23.039866 14742 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.039973 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.041718 14742 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.042456 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.042662 14742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.042748 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "9a89114615bd4372ab6281a3554fa6d9"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "af78354251c318260c1d8b0741929c56"
server_key_iv: "801721ccb20b27cf5bcc0bb19229da57"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.043073 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.061472 14742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.063532 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.063651 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.063877 14742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.064304 14784 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:39:23.066198 14742 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:39:23.066251 14742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:39:23.066283 14742 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:39:23.066890 14742 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:39:23.066920 14742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:39:23.067004 14784 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap starting.
I20260924 19:39:23.073256 14742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44359
I20260924 19:39:23.073374 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44359 every 8 connection(s)
I20260924 19:39:23.073637 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:39:23.077319 14784 log.cc:824] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:23.079746 14784 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:39:23.080040 14784 tablet_bootstrap.cc:492] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap complete.
I20260924 19:39:23.080685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14742
I20260924 19:39:23.081108 14784 ts_tablet_manager.cc:1397] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20260924 19:39:23.081248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42869
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=45929
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:23.082347 14784 raft_consensus.cc:348] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:23.082630 14784 raft_consensus.cc:729] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a89114615bd4372ab6281a3554fa6d9, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.082777 14784 consensus_queue.cc:260] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } } peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 19:39:23.083017 14784 ts_tablet_manager.cc:1428] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:39:23.087118 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.074299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:59919)
0924 19:39:23.076321 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.076354 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.076780 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.077247 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.077251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.077579 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:23.077839 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.077845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.078944 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.078945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.079738 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.079740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.080437 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.080438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.080564 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.081132 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:23.081142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:23.083216 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:23.085365 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.085373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.085375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.085659 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.085935 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.085936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.085936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.086067 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.086473 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:23.086479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:23.086635 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.086748 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.086915 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1987,"thread_start_us":89,"threads_started":1}
I20260924 19:39:23.087199 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.074371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59919 (local address 127.0.98.62:44217)
0924 19:39:23.076925 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.076932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.076958 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:39:23.077053 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.077057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.077093 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.077206 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:23.077948 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.078839 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.079850 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.080343 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.080440 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.083337 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:23.083341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:23.083341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:23.083354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:23.085266 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.085740 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.085743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.085744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.085867 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.086129 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.086130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.086130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.086403 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:23.086485 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.086818 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.086965 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2513,"thread_start_us":60,"threads_started":1}
I20260924 19:39:23.087792 14892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:23.087900 14892 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:23.088104 14892 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:23.088865 14704 ts_manager.cc:194] Registered new tserver with Master: 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359)
I20260924 19:39:23.090014 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59919
I20260924 19:39:23.091001 14892 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:39:23.098506 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.090298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.98.1:44359)
0924 19:39:23.090445 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.090449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.090458 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:23.090489 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.090490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.090503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.091017 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:23.091139 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.092209 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.093322 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.093730 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.094065 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.095839 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:23.095846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:23.095846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:23.095852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:23.097421 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.097763 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.097766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.097766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.097814 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.097984 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.097985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.097985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.098104 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:23.098176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.098338 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.098415 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260924 19:39:23.098513 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.089463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44359 (local address 127.0.0.1:51654)
0924 19:39:23.089698 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.089727 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.089781 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.090590 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.090595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.090892 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:23.090946 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.090948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.092296 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.092297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.093231 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.093233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.093819 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.093823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.093889 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.094344 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:23.094349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:23.095757 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:23.097503 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.097508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.097510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.097692 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.097879 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.097880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.097880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.097935 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.098166 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:23.098172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:23.098220 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.098304 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.098415 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260924 19:39:23.098949 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ce26148c7846a1b6e9e43b428273b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260924 19:39:23.099498 14901 tablet_replica.cc:330] stopping tablet replica
I20260924 19:39:23.099591 14901 raft_consensus.cc:2229] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:23.099650 14901 raft_consensus.cc:2258] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:23.099972 14901 ts_tablet_manager.cc:1905] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:23.101615 14901 ts_tablet_manager.cc:1918] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:23.101673 14901 log.cc:1196] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/d3ce26148c7846a1b6e9e43b428273b2
I20260924 19:39:23.101842 14901 ts_tablet_manager.cc:1939] T d3ce26148c7846a1b6e9e43b428273b2 P 9a89114615bd4372ab6281a3554fa6d9: Deleting consensus metadata
I20260924 19:39:23.102247 14691 catalog_manager.cc:4841] TS 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359): tablet d3ce26148c7846a1b6e9e43b428273b2 (table test-table [id=66c24a92526649d18401cbbe0d149243]) successfully deleted
W20260924 19:39:23.162515 14897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.162703 14897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.162725 14897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.163834 14897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.163872 14897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.163897 14897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:23.165373 14897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45929
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.165582 14897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.165791 14897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.170475 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.170469 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.170578 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.170810 14897 server_base.cc:1034] running on GCE node
I20260924 19:39:23.170955 14897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.171170 14897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.172297 14897 hybrid_clock.cc:648] HybridClock initialized: now 1790278763172286 us; error 27 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.174517 14897 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:23.174891 14897 webserver.cc:460] Webserver started at http://127.0.98.2:45929/ using document root <none> and password file <none>
I20260924 19:39:23.175091 14897 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.175148 14897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.176474 14897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.177011 14916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.177145 14897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:23.177227 14897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "5b93a0bd72564816af9efc67efc08959"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "00605bbec0bb3e1b89c8f5825fe8b151"
server_key_iv: "a77ac739021199e065eee7cb8623cac2"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.177520 14897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.185119 14897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.187045 14897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.187147 14897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.187316 14897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.187541 14897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:23.187566 14897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.187587 14897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:23.187603 14897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.192780 14897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42869
I20260924 19:39:23.192847 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42869 every 8 connection(s)
I20260924 19:39:23.193111 14897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:39:23.196456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 14897
I20260924 19:39:23.196762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35243
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=39701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:23.202879 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.193773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48593 (local address 127.0.98.62:44217)
0924 19:39:23.193826 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.193829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.194017 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:39:23.194542 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.194543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.194564 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.194614 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:23.195519 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.196081 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.197438 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.197686 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.197768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.200097 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:23.200098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:23.200099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:23.200105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:23.201588 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.201958 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.201959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.201959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.202026 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.202246 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.202247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.202247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.202394 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:23.202473 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.202682 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.202776 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:23.202898 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.193732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:48593)
0924 19:39:23.193902 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.193937 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.194438 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.194696 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.194699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.195237 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:23.195418 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.195423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.196162 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.196163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.197353 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.197357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.197783 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.197784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.197928 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.198402 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:23.198411 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:23.200007 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:23.201673 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.201679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.201681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.201884 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.202091 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.202092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.202092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.202173 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.202459 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:23.202462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:23.202567 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.202642 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.202751 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 19:39:23.203330 15030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:23.203415 15030 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:23.203575 15030 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:23.203976 14704 ts_manager.cc:194] Registered new tserver with Master: 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869)
I20260924 19:39:23.204442 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48593
W20260924 19:39:23.278091 15033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.278316 15033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.278342 15033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.279440 15033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.279479 15033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.279505 15033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:23.281060 15033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39701
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.281280 15033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.281471 15033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.286366 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.286378 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.286558 15033 server_base.cc:1034] running on GCE node
W20260924 19:39:23.286378 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.286808 15033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.287029 15033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.288189 15033 hybrid_clock.cc:648] HybridClock initialized: now 1790278763288173 us; error 26 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.290359 15033 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:23.290784 15033 webserver.cc:460] Webserver started at http://127.0.98.3:39701/ using document root <none> and password file <none>
I20260924 19:39:23.290979 15033 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.291041 15033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.292387 15033 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.292928 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.293057 15033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:23.293154 15033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "e3a79941bd0061b3a2f01460ce3a7946"
server_key_iv: "5b3f3b6e178d5755fe69fcf98098e59d"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.293481 15033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.306706 15033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.308629 15033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.308765 15033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.308974 15033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.309262 15033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:23.309293 15033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.309321 15033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:23.309345 15033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.314659 15033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35243
I20260924 19:39:23.314708 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35243 every 8 connection(s)
I20260924 19:39:23.315014 15033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:23.321885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15033
I20260924 19:39:23.323421 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.315660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38525 (local address 127.0.98.62:44217)
0924 19:39:23.315721 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.315724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.315902 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:39:23.316420 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.316421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.316430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.316471 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:23.317360 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.317851 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.318792 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.318969 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.319018 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.320949 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:23.320950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:23.320950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:23.320956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:23.322127 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.322500 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.322501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.322501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.322560 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.322805 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.322806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.322806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.322904 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:23.322950 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.323253 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.323328 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:39:23.323530 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.315636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:38525)
0924 19:39:23.315785 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.315826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.316322 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.316528 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.316531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.317062 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:23.317254 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.317259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.317930 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.317931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.318727 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.318729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.319023 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.319024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.319113 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.319538 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:23.319546 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:23.320872 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:23.322209 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.322216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.322219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.322452 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.322613 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:23.322613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:23.322613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:23.322748 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:23.322955 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:23.322960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:23.323110 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.323208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.323322 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 19:39:23.323975 15162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44217
I20260924 19:39:23.324069 15162 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:23.324239 15162 heartbeater.cc:511] Master 127.0.98.62:44217 requested a full tablet report, sending...
I20260924 19:39:23.324651 14704 ts_manager.cc:194] Registered new tserver with Master: 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
I20260924 19:39:23.325021 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38525
I20260924 19:39:23.328325 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.322221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.0.98.62:44217)
0924 19:39:23.322311 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.322313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.322392 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:39:23.322454 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.322454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.322465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.322522 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:23.323129 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.323805 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.324851 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.325043 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.325120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.326207 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:23.326208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:23.326208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:23.326215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:23.327323 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.327622 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.327623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.327623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.327675 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:23.327831 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:23.327832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:23.327832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:23.327952 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:23.328099 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.328158 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.328237 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
I20260924 19:39:23.328999 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:39:23.331408 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.329288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.0.98.62:44217)
0924 19:39:23.329341 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.329342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.329405 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:39:23.329458 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.329458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.329482 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.329533 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:39:23.329638 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.330058 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.330787 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.330969 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.331021 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.331126 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:39:23.331214 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:39:23.331266 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.331342 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.331358 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:39:23.331627 14704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:39:23.333140 14704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:39:23.335126 14816 tablet_service.cc:1467] Processing CreateTablet for tablet 47f722aef2164127b22ee243c57f8c4d (DEFAULT_TABLE table=test-table [id=7277bfe284ac4bca9607b245e5da4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:23.335328 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f722aef2164127b22ee243c57f8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:23.336580 14784 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap starting.
I20260924 19:39:23.336897 14784 tablet_bootstrap.cc:654] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:23.337745 14784 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: No bootstrap required, opened a new log
I20260924 19:39:23.337793 14784 ts_tablet_manager.cc:1397] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.337944 14784 raft_consensus.cc:348] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.338019 14784 raft_consensus.cc:374] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:23.338069 14784 raft_consensus.cc:729] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a89114615bd4372ab6281a3554fa6d9, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.338111 14784 consensus_queue.cc:260] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.338195 14784 ts_tablet_manager.cc:1428] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.338657 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.334689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.0.1:33674)
0924 19:39:23.334728 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.334739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.334775 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.335268 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.335269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.335270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:23.335307 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.335309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.336669 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.336671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.337550 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.337553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.338406 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.338407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.338473 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.338490 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.338551 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.338591 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:23.338676 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.334943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.2:42869)
0924 19:39:23.335111 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.335115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.335124 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:23.335159 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.335161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.335174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.335376 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:23.335448 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.336552 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.337654 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.338336 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.338399 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.338458 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.338565 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.338603 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260924 19:39:23.338900 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.334747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.0.1:41742)
0924 19:39:23.334870 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.334878 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.334915 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.335205 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.335206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.335209 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:23.335249 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.335250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.336775 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.336776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.337763 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.337765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.338692 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.338693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.338751 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.338754 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.338798 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.338836 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 19:39:23.339112 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.334813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.98.3:35243)
0924 19:39:23.335036 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.335041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.335050 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:23.335085 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.335088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.335102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.335179 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:23.335342 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.336534 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.337857 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.338549 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.338606 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.338659 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.339001 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.339041 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260924 19:39:23.339320 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 47f722aef2164127b22ee243c57f8c4d (DEFAULT_TABLE table=test-table [id=7277bfe284ac4bca9607b245e5da4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:23.339601 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f722aef2164127b22ee243c57f8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:23.339635 15096 tablet_service.cc:1467] Processing CreateTablet for tablet 47f722aef2164127b22ee243c57f8c4d (DEFAULT_TABLE table=test-table [id=7277bfe284ac4bca9607b245e5da4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:23.339844 15096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f722aef2164127b22ee243c57f8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:23.342249 15172 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Bootstrap starting.
I20260924 19:39:23.342446 15173 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Bootstrap starting.
I20260924 19:39:23.342896 15172 tablet_bootstrap.cc:654] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:23.342960 15173 tablet_bootstrap.cc:654] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:23.343340 15172 log.cc:824] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:23.343365 15173 log.cc:824] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:23.344014 15172 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: No bootstrap required, opened a new log
I20260924 19:39:23.344091 15172 ts_tablet_manager.cc:1397] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:23.344255 15173 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: No bootstrap required, opened a new log
I20260924 19:39:23.344318 15173 ts_tablet_manager.cc:1397] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:23.345943 15172 raft_consensus.cc:348] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.346134 15172 raft_consensus.cc:374] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:23.346181 15172 raft_consensus.cc:729] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93a0bd72564816af9efc67efc08959, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.346138 15173 raft_consensus.cc:348] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.346279 15173 raft_consensus.cc:374] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:23.346316 15173 raft_consensus.cc:729] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.346299 15172 consensus_queue.cc:260] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.346410 15173 consensus_queue.cc:260] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.346583 15030 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
I20260924 19:39:23.346568 15172 ts_tablet_manager.cc:1428] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:39:23.346658 15173 ts_tablet_manager.cc:1428] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:39:23.346704 15162 heartbeater.cc:503] Master 127.0.98.62:44217 was elected leader, sending a full tablet report...
I20260924 19:39:23.360833 15176 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:23.360922 15176 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.361195 15176 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243), 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359)
I20260924 19:39:23.364030 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.361480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51047 (local address 127.0.98.3:35243)
0924 19:39:23.361536 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.361538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.361547 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:39:23.361567 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.361568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.361576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.361627 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:23.361763 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.362214 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363286 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.363686 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363738 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.363745 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.363899 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.363969 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:39:23.364020 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.361328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.98.2:51047)
0924 19:39:23.361370 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.361380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.361418 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.361638 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.361639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.361641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:23.361711 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.361713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.362296 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.362297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.363173 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.363176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.363754 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.363832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.363835 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.363884 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.363923 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:23.364054 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.361494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51573 (local address 127.0.98.1:44359)
0924 19:39:23.361538 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.361540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.361610 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:39:23.361667 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.361668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.361688 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.361748 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:23.361868 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.362320 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363323 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.363747 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363814 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.363889 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.363957 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.363999 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:39:23.364109 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.361421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44359 (local address 127.0.98.2:51573)
0924 19:39:23.361560 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.361568 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.361610 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.361755 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:23.361755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:23.361757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:23.361815 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.361817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.362396 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.362397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.363256 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:23.363259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.363815 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.363815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:23.363868 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.363872 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:39:23.363923 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:39:23.363978 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260924 19:39:23.364413 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f722aef2164127b22ee243c57f8c4d" candidate_uuid: "5b93a0bd72564816af9efc67efc08959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a89114615bd4372ab6281a3554fa6d9" is_pre_election: true
I20260924 19:39:23.364439 15116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f722aef2164127b22ee243c57f8c4d" candidate_uuid: "5b93a0bd72564816af9efc67efc08959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" is_pre_election: true
I20260924 19:39:23.364544 15116 raft_consensus.cc:2454] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93a0bd72564816af9efc67efc08959 in term 0.
I20260924 19:39:23.364537 14840 raft_consensus.cc:2454] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b93a0bd72564816af9efc67efc08959 in term 0.
I20260924 19:39:23.364734 14920 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959; no voters:
I20260924 19:39:23.364876 15176 raft_consensus.cc:2781] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:39:23.364926 15176 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:39:23.364948 15176 raft_consensus.cc:3037] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:23.365613 15176 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.365725 15176 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [CANDIDATE]: Term 1 election: Requested vote from peers 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243), 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359)
I20260924 19:39:23.365883 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f722aef2164127b22ee243c57f8c4d" candidate_uuid: "5b93a0bd72564816af9efc67efc08959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a89114615bd4372ab6281a3554fa6d9"
I20260924 19:39:23.365891 15116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f722aef2164127b22ee243c57f8c4d" candidate_uuid: "5b93a0bd72564816af9efc67efc08959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
I20260924 19:39:23.365950 15116 raft_consensus.cc:3037] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:23.365953 14840 raft_consensus.cc:3037] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:23.366622 14840 raft_consensus.cc:2454] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93a0bd72564816af9efc67efc08959 in term 1.
I20260924 19:39:23.366680 15116 raft_consensus.cc:2454] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b93a0bd72564816af9efc67efc08959 in term 1.
I20260924 19:39:23.366770 14920 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b93a0bd72564816af9efc67efc08959, 9a89114615bd4372ab6281a3554fa6d9; no voters:
I20260924 19:39:23.366886 15176 raft_consensus.cc:2781] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:23.367028 15176 raft_consensus.cc:686] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 1 LEADER]: Becoming Leader. State: Replica: 5b93a0bd72564816af9efc67efc08959, State: Running, Role: LEADER
I20260924 19:39:23.367112 15176 consensus_queue.cc:237] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.367823 14704 catalog_manager.cc:5495] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b93a0bd72564816af9efc67efc08959 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5b93a0bd72564816af9efc67efc08959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } health_report { overall_health: UNKNOWN } } }
I20260924 19:39:23.374459 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.372592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.0.98.2:42869)
0924 19:39:23.372637 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:23.372638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:23.372645 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:39:23.372714 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:23.372714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:23.372728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:23.372780 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:39:23.372855 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.373270 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.373921 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:23.374086 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:23.374125 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:23.374237 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:39:23.374318 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:39:23.374368 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:23.374382 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 19:39:23.374399 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:39:23.375914 14840 raft_consensus.cc:1261] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Refusing update from remote peer 5b93a0bd72564816af9efc67efc08959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:23.375910 15116 raft_consensus.cc:1261] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Refusing update from remote peer 5b93a0bd72564816af9efc67efc08959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:23.376220 15179 consensus_queue.cc:1035] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:23.376338 15176 consensus_queue.cc:1035] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:23.378324 15182 mvcc.cc:205] Tried to move back new op lower bound from 7332981814785568768 to 7332981814752374784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981814785568768}]
I20260924 19:39:23.378443 15184 mvcc.cc:205] Tried to move back new op lower bound from 7332981814785568768 to 7332981814752374784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981814785568768}]
I20260924 19:39:23.378484 15183 mvcc.cc:205] Tried to move back new op lower bound from 7332981814785568768 to 7332981814752374784. Current Snapshot: MvccSnapshot[applied={T|T < 7332981814785568768}]
I20260924 19:39:23.380455 14704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:23.380527 14704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:23.381127 14704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Sending DeleteTablet for 3 replicas of tablet 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:23.381377 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f722aef2164127b22ee243c57f8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260924 19:39:23.381417 15096 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f722aef2164127b22ee243c57f8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41742
I20260924 19:39:23.381476 14901 tablet_replica.cc:330] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: stopping tablet replica
I20260924 19:39:23.381546 14901 raft_consensus.cc:2229] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:23.381603 14901 raft_consensus.cc:2258] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:23.381604 15192 tablet_replica.cc:330] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: stopping tablet replica
I20260924 19:39:23.381693 15192 raft_consensus.cc:2229] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:23.381691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14742
I20260924 19:39:23.381790 15192 raft_consensus.cc:2258] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:23.381897 14901 ts_tablet_manager.cc:1905] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:23.382164 15192 ts_tablet_manager.cc:1905] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:23.383622 15192 ts_tablet_manager.cc:1918] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:23.383704 15192 log.cc:1196] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:23.384095 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f722aef2164127b22ee243c57f8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260924 19:39:23.384261 15192 ts_tablet_manager.cc:1939] T 47f722aef2164127b22ee243c57f8c4d P 2e2b6f40ac8e4fdd8e6af88b80e5dccb: Deleting consensus metadata
I20260924 19:39:23.384635 15193 tablet_replica.cc:330] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: stopping tablet replica
I20260924 19:39:23.384657 14691 catalog_manager.cc:4841] TS 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): tablet 47f722aef2164127b22ee243c57f8c4d (table test-table [id=7277bfe284ac4bca9607b245e5da4969]) successfully deleted
I20260924 19:39:23.384737 15193 raft_consensus.cc:2229] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:39:23.384847 15193 raft_consensus.cc:2258] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:23.385228 15193 ts_tablet_manager.cc:1905] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:23.386663 15193 ts_tablet_manager.cc:1918] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:23.386729 15193 log.cc:1196] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:23.386803 14691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44359 (error 108)
I20260924 19:39:23.387084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14897
I20260924 19:39:23.387097 15193 ts_tablet_manager.cc:1939] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Deleting consensus metadata
I20260924 19:39:23.387439 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.387349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44359 (local address 127.0.0.1:51668)
0924 19:39:23.387386 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.387411 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 19:39:23.387589 14691 catalog_manager.cc:4584] TS 9a89114615bd4372ab6281a3554fa6d9 (127.0.98.1:44359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47f722aef2164127b22ee243c57f8c4d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44359: connect: Connection refused (error 111)
I20260924 19:39:23.391620 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.391526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.0.1:33696)
0924 19:39:23.391565 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.391592 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:23.391646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15033
I20260924 19:39:23.395429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 14671
I20260924 19:39:23.399814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--webserver_interface=127.0.98.62
--webserver_port=43017
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:23.476600 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.476797 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.476826 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.477918 15195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:23.477954 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.477972 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.477988 15195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:23.478003 15195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:23.479408 15195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.479630 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.479820 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.484198 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.484171 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.484259 15195 server_base.cc:1034] running on GCE node
W20260924 19:39:23.484180 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.484639 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.484876 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.486011 15195 hybrid_clock.cc:648] HybridClock initialized: now 1790278763485996 us; error 29 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.488276 15195 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:23.488656 15195 webserver.cc:460] Webserver started at http://127.0.98.62:43017/ using document root <none> and password file <none>
I20260924 19:39:23.488829 15195 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.488867 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.490016 15195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:23.490557 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.490712 15195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.490783 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "602e3421ca8e42da80818268c95a7f6e"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "8ff4a6f45ef25e55a0169d6323458c15"
server_key_iv: "4da113e7a25634ac57061b3cdee15157"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.491107 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.505692 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.507258 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.507372 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.510946 15195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44217
I20260924 19:39:23.510974 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44217 every 8 connection(s)
I20260924 19:39:23.511309 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:23.512763 15263 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:39:23.513365 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap starting.
I20260924 19:39:23.513711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15195
I20260924 19:39:23.514009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44359
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=38915
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:39:23.521373 15263 log.cc:824] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:23.523346 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:39:23.523582 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e: Bootstrap complete.
I20260924 19:39:23.525542 15263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:23.525758 15263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.525889 15263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:23.525941 15263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:23.525985 15263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:23.526027 15263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:39:23.526923 15263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:23.527014 15263 leader_election.cc:302] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602e3421ca8e42da80818268c95a7f6e; no voters:
I20260924 19:39:23.527169 15263 leader_election.cc:288] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:39:23.527221 15267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:39:23.527331 15267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [term 3 LEADER]: Becoming Leader. State: Replica: 602e3421ca8e42da80818268c95a7f6e, State: Running, Role: LEADER
I20260924 19:39:23.527412 15263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:23.527412 15267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } }
I20260924 19:39:23.527738 15268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:23.527812 15268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:23.528097 15269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602e3421ca8e42da80818268c95a7f6e. Latest consensus state: current_term: 3 leader_uuid: "602e3421ca8e42da80818268c95a7f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602e3421ca8e42da80818268c95a7f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44217 } } }
I20260924 19:39:23.528165 15269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602e3421ca8e42da80818268c95a7f6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:23.528316 15273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:39:23.529040 15273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:23.529129 15273 catalog_manager.cc:1259] Loaded cluster ID: 6c8e1fdc768c48719a0f13baf55145a1
I20260924 19:39:23.529163 15273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 19:39:23.529371 15273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a74c129ad google::LogMessage::Fail() at ??:0
@ 0x7f6a74c1588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a74c124de google::LogMessage::Flush() at ??:0
@ 0x7f6a74c13a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a7cb330f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6a7cb3c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a76ae00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a76ad5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a779396db start_thread at ??:0
@ 0x7f6a7371061f clone at ??:0
W20260924 19:39:23.603591 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.603783 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.603804 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.604947 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.604985 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.605011 15265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:23.606420 15265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38915
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.606640 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.606845 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.612474 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.612473 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.612740 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.612802 15265 server_base.cc:1034] running on GCE node
I20260924 19:39:23.612991 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.613210 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.614359 15265 hybrid_clock.cc:648] HybridClock initialized: now 1790278763614349 us; error 24 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.616768 15265 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:23.617156 15265 webserver.cc:460] Webserver started at http://127.0.98.1:38915/ using document root <none> and password file <none>
I20260924 19:39:23.617344 15265 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.617390 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.618675 15265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:23.619378 15310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.619540 15265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:23.619626 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "9a89114615bd4372ab6281a3554fa6d9"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "af78354251c318260c1d8b0741929c56"
server_key_iv: "801721ccb20b27cf5bcc0bb19229da57"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.619971 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.632755 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.634683 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.634809 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.635005 15265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.635389 15317 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:39:23.637287 15265 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:39:23.637343 15265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:39:23.637378 15265 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:39:23.637971 15265 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:39:23.638003 15265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:39:23.638103 15317 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap starting.
I20260924 19:39:23.639890 15317 log.cc:824] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:23.642721 15317 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:39:23.642992 15317 tablet_bootstrap.cc:492] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Bootstrap complete.
I20260924 19:39:23.643132 15317 ts_tablet_manager.cc:1397] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:39:23.644516 15265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44359
I20260924 19:39:23.644603 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44359 every 8 connection(s)
I20260924 19:39:23.644847 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:39:23.644891 15317 raft_consensus.cc:348] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.645342 15317 raft_consensus.cc:729] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a89114615bd4372ab6281a3554fa6d9, State: Initialized, Role: FOLLOWER
I20260924 19:39:23.645463 15317 consensus_queue.cc:260] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:23.645715 15317 ts_tablet_manager.cc:1428] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9: Time spent starting tablet: real 0.003s user 0.004s sys 0.001s
I20260924 19:39:23.649065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15265
I20260924 19:39:23.649305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42869
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=45929
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:23.710924 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:39:23.645500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:60695)
0924 19:39:23.645674 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.645716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:23.645859 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:23.710700 (+ 64841us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
W20260924 19:39:23.711202 15426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44217 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44217: Client connection negotiation failed: client connection to 127.0.98.62:44217: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:39:23.711537 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.711418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:51953)
0924 19:39:23.711471 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.711512 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:39:23.711896 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.711814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:36971)
0924 19:39:23.711854 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.711878 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:39:23.712016 15426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:39:23.729063 15430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.729276 15430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.729300 15430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.730466 15430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.730509 15430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.730537 15430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:23.732035 15430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45929
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.732255 15430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.732456 15430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.737221 15438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.737238 15435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.737242 15436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.737406 15430 server_base.cc:1034] running on GCE node
I20260924 19:39:23.737663 15430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.737892 15430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.739041 15430 hybrid_clock.cc:648] HybridClock initialized: now 1790278763739017 us; error 37 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.741433 15430 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:23.741823 15430 webserver.cc:460] Webserver started at http://127.0.98.2:45929/ using document root <none> and password file <none>
I20260924 19:39:23.742025 15430 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.742075 15430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.743270 15430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:39:23.743760 15445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.743913 15430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.744014 15430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "5b93a0bd72564816af9efc67efc08959"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "00605bbec0bb3e1b89c8f5825fe8b151"
server_key_iv: "a77ac739021199e065eee7cb8623cac2"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.744356 15430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.751912 15430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.753859 15430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.753965 15430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.754132 15430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.754446 15452 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 19:39:23.755685 15452 ts_tablet_manager.cc:1852] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 19:39:23.756029 15452 ts_tablet_manager.cc:1905] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:23.757905 15452 ts_tablet_manager.cc:1918] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:23.757992 15452 ts_tablet_manager.cc:1939] T 47f722aef2164127b22ee243c57f8c4d P 5b93a0bd72564816af9efc67efc08959: Deleting consensus metadata
I20260924 19:39:23.758201 15430 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 19:39:23.758249 15430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 19:39:23.758293 15430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:23.758319 15430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.763841 15430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42869
I20260924 19:39:23.763918 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42869 every 8 connection(s)
I20260924 19:39:23.764182 15430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:39:23.765133 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.764746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:50417)
0924 19:39:23.764887 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.764953 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260924 19:39:23.765296 15560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44217 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44217: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
I20260924 19:39:23.765497 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.765436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:40699)
0924 19:39:23.765461 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.765483 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 19:39:23.765734 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.765688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:43523)
0924 19:39:23.765709 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.765725 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 19:39:23.765828 15560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:39:23.773819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15430
I20260924 19:39:23.774053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35243
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44217
--webserver_port=39701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:23.852221 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:23.852413 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:23.852443 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:23.853560 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:23.853597 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:23.853631 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:23.855026 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39701
--tserver_master_addrs=127.0.98.62:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:23.855257 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:23.855459 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:23.860009 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:23.860015 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.860188 15563 server_base.cc:1034] running on GCE node
W20260924 19:39:23.860278 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:23.860460 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:23.860697 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:23.861833 15563 hybrid_clock.cc:648] HybridClock initialized: now 1790278763861823 us; error 29 us; skew 500 ppm
Sep 24 19:39:23 dist-test-slave-6kf3 krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:23.864233 15563 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:23.864687 15563 webserver.cc:460] Webserver started at http://127.0.98.3:39701/ using document root <none> and password file <none>
I20260924 19:39:23.864899 15563 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:23.864950 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:23.866226 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:39:23.866768 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.866914 15563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:23.866995 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb"
format_stamp: "Formatted at 2026-09-24 19:39:22 on dist-test-slave-6kf3"
server_key: "e3a79941bd0061b3a2f01460ce3a7946"
server_key_iv: "5b3f3b6e178d5755fe69fcf98098e59d"
server_key_version: "encryptionkey@0"
I20260924 19:39:23.867321 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:23.904484 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:23.906570 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:23.906698 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:23.906903 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:23.907176 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:23.907207 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.907235 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:23.907259 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:23.912689 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35243
I20260924 19:39:23.912746 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35243 every 8 connection(s)
I20260924 19:39:23.913048 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:39:23.913941 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.913610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:58431)
0924 19:39:23.913744 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.913809 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260924 19:39:23.914301 15692 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44217 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44217: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
I20260924 19:39:23.914534 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.914457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:35097)
0924 19:39:23.914495 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.914517 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:39:23.914753 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:23.914684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:48269)
0924 19:39:23.914715 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:23.914735 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:39:23.914835 15692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:39:23.918422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15563
I20260924 19:39:24.712563 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.712325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:43017)
0924 19:39:24.712451 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:24.712497 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260924 19:39:24.766311 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.766091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:38483)
0924 19:39:24.766218 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:24.766262 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260924 19:39:24.790369 15698 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:24.790474 15698 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:24.790805 15698 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
Sep 24 19:39:24 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 19:39:24 dist-test-slave-6kf3 krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 19:39:24.800218 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.791061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57435 (local address 127.0.98.2:42869)
0924 19:39:24.791234 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:24.791242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:24.791263 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:39:24.791725 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:24.791727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:24.791763 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:24.791891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:24.792637 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.793459 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794449 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.794828 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:24.796939 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:24.796942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:24.796942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:24.796951 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:24.798571 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799104 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:24.799107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:24.799107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:24.799189 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799416 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:24.799417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:24.799418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:24.799632 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:24.799687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:24.799994 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:39:24.800109 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 19:39:24.800225 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.791034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49083 (local address 127.0.98.3:35243)
0924 19:39:24.791179 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:24.791192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:24.791217 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:39:24.791649 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:24.791651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:24.791705 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:24.791811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:24.792569 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.793466 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794453 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.794838 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794938 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:24.797316 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:24.797319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:24.797320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:24.797329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:24.798707 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799104 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:24.799107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:24.799107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:24.799190 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799415 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:24.799416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:24.799416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:24.799673 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:24.799748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:24.800014 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:39:24.800129 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
I20260924 19:39:24.800232 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.790921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.98.1:49083)
0924 19:39:24.791115 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:24.791124 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:24.791175 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:24.791887 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:24.791890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:24.792258 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:24.792450 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:24.792455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.793563 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.793563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:24.794373 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:24.794375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.794906 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:24.795016 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:24.795502 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:24.795509 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:24.797226 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:24.798772 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:24.798774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:24.798774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:24.799035 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:24.799248 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:24.799249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:24.799360 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:24.799742 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:24.799743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:24.799827 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 19:39:24.799910 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:39:24.800102 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":131,"thread_start_us":53,"threads_started":1}
I20260924 19:39:24.800565 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.790920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.98.1:57435)
0924 19:39:24.790975 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:24.790987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:24.791050 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:24.791892 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:24.791893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:24.792477 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:24.792544 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:24.792546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.793549 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.793550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:24.794375 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:24.794377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:24.794904 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:24.794905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:24.795019 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:24.795559 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:24.795563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:24.796843 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:24.798660 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:24.798666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:24.798669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:24.799035 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:24.799249 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:24.799250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:24.799250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:24.799362 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:24.799694 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:24.799700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:24.799834 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:39:24.799910 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:39:24.800172 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":156}
W20260924 19:39:24.801012 15314 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:24.801402 15311 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:24.801441 15311 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a89114615bd4372ab6281a3554fa6d9; no voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959
I20260924 19:39:24.801545 15698 raft_consensus.cc:2726] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:39:24.915278 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:24.915094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:49385)
0924 19:39:24.915205 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:24.915244 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260924 19:39:25.713165 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:25.712959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:33121)
0924 19:39:25.713079 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:25.713121 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 19:39:25.766973 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:25.766789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:42555)
0924 19:39:25.766902 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:25.766936 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260924 19:39:25.915829 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:25.915639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:48897)
0924 19:39:25.915759 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:25.915794 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260924 19:39:26.713755 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:26.713566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:34425)
0924 19:39:26.713679 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:26.713710 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 19:39:26.764170 15707 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:26.764256 15707 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:26.764385 15707 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
W20260924 19:39:26.764724 15311 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:26.764804 15314 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:26.764858 15314 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a89114615bd4372ab6281a3554fa6d9; no voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959
I20260924 19:39:26.764943 15707 raft_consensus.cc:2726] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:39:26.767545 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:26.767376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:60039)
0924 19:39:26.767478 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:26.767513 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 19:39:26.916497 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:26.916269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:57977)
0924 19:39:26.916397 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:26.916432 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260924 19:39:27.714424 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:27.714192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:34427)
0924 19:39:27.714338 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:27.714378 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260924 19:39:27.768124 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:27.767906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:49029)
0924 19:39:27.768050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:27.768085 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260924 19:39:27.917076 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:27.916888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:55277)
0924 19:39:27.916994 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:27.917031 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260924 19:39:28.393599 15713 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:28.393690 15713 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:28.393817 15713 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
W20260924 19:39:28.394155 15314 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:28.394238 15311 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:28.394307 15311 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a89114615bd4372ab6281a3554fa6d9; no voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959
I20260924 19:39:28.394418 15713 raft_consensus.cc:2726] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:39:28.715147 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:28.714940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:56313)
0924 19:39:28.715067 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:28.715098 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 19:39:28.768663 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:28.768469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:45243)
0924 19:39:28.768593 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:28.768626 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260924 19:39:28.917661 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:28.917431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:43983)
0924 19:39:28.917543 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:28.917619 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260924 19:39:29.715834 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:29.715590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:33241)
0924 19:39:29.715753 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:29.715788 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260924 19:39:29.769251 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:29.769056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:51937)
0924 19:39:29.769175 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:29.769211 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260924 19:39:29.918227 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:29.918043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:55785)
0924 19:39:29.918155 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:29.918189 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260924 19:39:30.716573 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:30.716305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:40673)
0924 19:39:30.716483 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:30.716527 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260924 19:39:30.769886 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:30.769678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:35829)
0924 19:39:30.769803 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:30.769844 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 19:39:30.918886 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:30.918669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:34871)
0924 19:39:30.918800 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:30.918841 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260924 19:39:31.114040 15723 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:31.114142 15723 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:31.114303 15723 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
W20260924 19:39:31.114669 15314 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:31.114743 15311 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:31.114780 15311 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a89114615bd4372ab6281a3554fa6d9; no voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959
I20260924 19:39:31.114850 15723 raft_consensus.cc:2726] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:39:31.717306 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:31.717070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:55627)
0924 19:39:31.717207 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:31.717259 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260924 19:39:31.770466 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:31.770277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:50421)
0924 19:39:31.770393 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:31.770426 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260924 19:39:31.919517 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:31.919299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:35683)
0924 19:39:31.919440 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:31.919476 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260924 19:39:32.718024 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:32.717787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:53125)
0924 19:39:32.717938 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:32.717979 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260924 19:39:32.771111 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:32.770870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:41029)
0924 19:39:32.771003 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:32.771047 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260924 19:39:32.920197 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:32.919976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:56385)
0924 19:39:32.920103 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:32.920148 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260924 19:39:33.718832 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:33.718531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:51325)
0924 19:39:33.718748 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:33.718788 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":130,"threads_started":1}
I20260924 19:39:33.771750 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:33.771522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:46301)
0924 19:39:33.771674 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:33.771708 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260924 19:39:33.920805 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:33.920567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:56105)
0924 19:39:33.920715 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:33.920764 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260924 19:39:34.719687 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:34.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:55835)
0924 19:39:34.719595 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:34.719642 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260924 19:39:34.733403 15734 raft_consensus.cc:482] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:34.733505 15734 raft_consensus.cc:504] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b93a0bd72564816af9efc67efc08959" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "2e2b6f40ac8e4fdd8e6af88b80e5dccb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } } peers { permanent_uuid: "9a89114615bd4372ab6281a3554fa6d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44359 } }
I20260924 19:39:34.733740 15734 leader_election.cc:288] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869), 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243)
W20260924 19:39:34.734124 15314 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e2b6f40ac8e4fdd8e6af88b80e5dccb (127.0.98.3:35243): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
W20260924 19:39:34.734213 15311 leader_election.cc:341] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b93a0bd72564816af9efc67efc08959 (127.0.98.2:42869): Not found: Tablet not found: 47f722aef2164127b22ee243c57f8c4d
I20260924 19:39:34.734256 15311 leader_election.cc:302] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a89114615bd4372ab6281a3554fa6d9; no voters: 2e2b6f40ac8e4fdd8e6af88b80e5dccb, 5b93a0bd72564816af9efc67efc08959
I20260924 19:39:34.734319 15734 raft_consensus.cc:2726] T 47f722aef2164127b22ee243c57f8c4d P 9a89114615bd4372ab6281a3554fa6d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:39:34.772410 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:34.772190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:50285)
0924 19:39:34.772327 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:34.772369 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 19:39:34.921414 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:34.921205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:53085)
0924 19:39:34.921335 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:34.921369 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260924 19:39:35.720448 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:35.720241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:38005)
0924 19:39:35.720359 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:35.720401 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260924 19:39:35.773046 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:35.772838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:60953)
0924 19:39:35.772957 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:35.772997 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 19:39:35.922127 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:35.921874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:37381)
0924 19:39:35.922027 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:35.922077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260924 19:39:36.721133 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:36.720910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:50569)
0924 19:39:36.721035 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:36.721075 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260924 19:39:36.773660 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:36.773451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:42969)
0924 19:39:36.773578 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:36.773613 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 19:39:36.922848 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:36.922583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:52957)
0924 19:39:36.922774 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:36.922807 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260924 19:39:37.721812 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:37.721557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:38865)
0924 19:39:37.721725 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:37.721766 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
I20260924 19:39:37.774366 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:37.774124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:50945)
0924 19:39:37.774275 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:37.774313 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260924 19:39:37.923420 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:37.923192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:48643)
0924 19:39:37.923326 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:37.923358 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260924 19:39:38.722648 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:38.722377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.1:39233)
0924 19:39:38.722534 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:38.722593 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 19:39:38.775022 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:38.774811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.2:52935)
0924 19:39:38.774941 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:38.774979 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260924 19:39:38.920128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15265
I20260924 19:39:38.924044 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:38.923848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44217 (local address 127.0.98.3:42341)
0924 19:39:38.923974 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:38.924006 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260924 19:39:38.924494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15430
I20260924 19:39:38.927922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15563
2026-09-24T19:39:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17934 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:39:38 dist-test-slave-6kf3 krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:39:38 dist-test-slave-6kf3 krb5kdc[15751](info): set up 2 sockets
Sep 24 19:39:38 dist-test-slave-6kf3 krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260924 19:39:39.876677 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:39:39 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:39:39.982847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:40.061887 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:40.062083 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:40.062104 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:40.063256 15764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:39:40.063297 15764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:39:40.063308 15764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:39:40.063319 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:39:40.063344 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:40.063375 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:40.063395 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:39:40.063416 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:39:40.064936 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:40.065168 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:40.065373 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:39:40.070230 15764 server_base.cc:1034] running on GCE node
W20260924 19:39:40.070282 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.070427 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.070369 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.070679 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:40.070957 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:40.072088 15764 hybrid_clock.cc:648] HybridClock initialized: now 1790278780072067 us; error 35 us; skew 500 ppm
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:40.074334 15764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:39:40.074769 15764 webserver.cc:460] Webserver started at http://127.0.98.62:37553/ using document root <none> and password file <none>
I20260924 19:39:40.074945 15764 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:40.074983 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:40.075071 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:40.076160 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "2b445e07a610482a92096698aa40eb2a"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "cd26356fe5a90d12a9a206f372a6c17e"
server_key_iv: "a6fc0bb8c8fab7576733395c37815ed0"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.076452 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b445e07a610482a92096698aa40eb2a"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "cd26356fe5a90d12a9a206f372a6c17e"
server_key_iv: "a6fc0bb8c8fab7576733395c37815ed0"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.077718 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:39:40.078418 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.078577 15764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:40.078655 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "2b445e07a610482a92096698aa40eb2a"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "cd26356fe5a90d12a9a206f372a6c17e"
server_key_iv: "a6fc0bb8c8fab7576733395c37815ed0"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.078712 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:40.098529 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:40.100301 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:40.100433 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:40.104105 15764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37969
I20260924 19:39:40.104144 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37969 every 8 connection(s)
I20260924 19:39:40.104460 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:39:40.105230 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:40.107007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15764
I20260924 19:39:40.107107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:39:40.107297 392 external_mini_cluster.cc:1428] Setting key e70c1f45cf83273883882cd9588ceb54
I20260924 19:39:40.107544 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap starting.
I20260924 19:39:40.108256 15832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:40.108567 15832 log.cc:824] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:40.109210 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: No bootstrap required, opened a new log
I20260924 19:39:40.110901 15832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:39:40.111032 15832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:40.111063 15832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Initialized, Role: FOLLOWER
I20260924 19:39:40.111183 15832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:39:40.111253 15832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:39:40.111299 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:39:40.111344 15832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:40.111822 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:39:40.111920 15832 leader_election.cc:302] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b445e07a610482a92096698aa40eb2a; no voters:
I20260924 19:39:40.112100 15832 leader_election.cc:288] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:39:40.112138 15837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:40.112305 15837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 LEADER]: Becoming Leader. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Running, Role: LEADER
I20260924 19:39:40.112404 15832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:39:40.112457 15837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:39:40.112864 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b445e07a610482a92096698aa40eb2a. Latest consensus state: current_term: 1 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:39:40.112939 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:39:40.113001 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:40.113023 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:39:40.113194 15846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:40.113929 15846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:39:40.116601 15846 catalog_manager.cc:1347] Generated new cluster ID: bd84d08babb14de497b09735c3e514df
I20260924 19:39:40.116654 15846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:39:40.118116 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.107779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.98.62:37969)
0924 19:39:40.107983 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.107992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.108027 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:39:40.108570 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.108579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.108649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.108801 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:40.109368 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.110254 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.111023 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.111553 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.111659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.114323 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:40.114328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:40.114329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:40.114340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:40.116183 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.116558 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.116561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.116562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.116664 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.116896 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.116897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.116897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.117192 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:40.117324 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.117771 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.117909 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260924 19:39:40.123577 15846 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:39:40.124210 15846 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:39:40.127170 15846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Generated new TSK 0
I20260924 19:39:40.127328 15846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:39:40.150645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:40.227725 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:40.227962 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:40.227998 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:40.229161 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:40.229204 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:40.229239 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:39:40.230675 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:40.230912 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:40.231115 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:40.235740 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.235759 15858 server_base.cc:1034] running on GCE node
W20260924 19:39:40.235991 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.235870 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.236240 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:40.236488 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:40.237628 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790278780237615 us; error 26 us; skew 500 ppm
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:40.239913 15858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:39:40.240352 15858 webserver.cc:460] Webserver started at http://127.0.98.1:39285/ using document root <none> and password file <none>
I20260924 19:39:40.240549 15858 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:40.240594 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:40.240713 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:40.241701 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "7494854bcfde48deaf02eb8f7bdc111c"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "0a8acfeeb25a078b238907f7f7ec0b5f"
server_key_iv: "934d214008e01014ff4a08740325a166"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.242022 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7494854bcfde48deaf02eb8f7bdc111c"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "0a8acfeeb25a078b238907f7f7ec0b5f"
server_key_iv: "934d214008e01014ff4a08740325a166"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.243302 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:39:40.243966 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.244107 15858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.244185 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "7494854bcfde48deaf02eb8f7bdc111c"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "0a8acfeeb25a078b238907f7f7ec0b5f"
server_key_iv: "934d214008e01014ff4a08740325a166"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.244249 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:40.259291 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:40.261338 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:40.261461 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:40.261665 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:40.261955 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:40.261991 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.262020 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:40.262042 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.267366 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45099
I20260924 19:39:40.267403 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45099 every 8 connection(s)
I20260924 19:39:40.267683 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:40.274796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15858
I20260924 19:39:40.274910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:39:40.275099 392 external_mini_cluster.cc:1428] Setting key 20a0e5c498702da109a32dddddc62175
I20260924 19:39:40.276201 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.268303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33485 (local address 127.0.98.62:37969)
0924 19:39:40.268363 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.268365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.268573 (+ 208us) server_negotiation.cc:407] Connection header received
0924 19:39:40.269057 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.269058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.269068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.269117 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:40.269971 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.270477 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.271416 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.271601 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.271646 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.273668 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:40.273670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:40.273670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:40.273678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:40.274828 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.275205 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.275207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.275207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.275274 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.275508 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.275509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.275510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.275639 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:40.275706 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.276064 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.276138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:39:40.276438 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.268305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:33485)
0924 19:39:40.268461 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.268496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.268978 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.269171 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.269175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.269699 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:40.269903 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.269908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.270542 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.270542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.271354 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.271356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.271655 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.271655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.271748 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.272295 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:40.272303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:40.273568 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:40.274900 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.274907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.274909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.275142 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.275338 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.275339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.275339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.275445 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.275702 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:40.275709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:40.275905 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.276051 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.276213 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 19:39:40.276949 15987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:39:40.277048 15987 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:40.277225 15987 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:39:40.277712 15796 ts_manager.cc:194] Registered new tserver with Master: 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
I20260924 19:39:40.278549 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:39:40.296454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:40.374661 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:40.374855 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:40.374877 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:40.376005 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:40.376042 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:40.376070 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:39:40.377458 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:40.377661 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:40.377858 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:40.382513 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.382498 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.382767 15992 server_base.cc:1034] running on GCE node
W20260924 19:39:40.382488 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.383049 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:40.383250 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:40.384402 15992 hybrid_clock.cc:648] HybridClock initialized: now 1790278780384385 us; error 33 us; skew 500 ppm
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:40.386591 15992 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:39:40.386987 15992 webserver.cc:460] Webserver started at http://127.0.98.2:35815/ using document root <none> and password file <none>
I20260924 19:39:40.387167 15992 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:40.387208 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:40.387333 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:40.388351 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "2e6037807dd14115bacd64760efb3e18"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "4fb2edb15c1006c98d6da8682d148971"
server_key_iv: "9fe69436a7794e71a0641ec6ded2e407"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.388659 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2e6037807dd14115bacd64760efb3e18"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "4fb2edb15c1006c98d6da8682d148971"
server_key_iv: "9fe69436a7794e71a0641ec6ded2e407"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.389801 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:39:40.390411 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.390535 15992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:39:40.390609 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "2e6037807dd14115bacd64760efb3e18"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "4fb2edb15c1006c98d6da8682d148971"
server_key_iv: "9fe69436a7794e71a0641ec6ded2e407"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.390669 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:40.397073 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:40.398573 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:40.398684 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:40.398865 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:40.399121 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:40.399152 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.399178 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:40.399201 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.404498 15992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260924 19:39:40.404547 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260924 19:39:40.404808 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:40.410509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 15992
I20260924 19:39:40.410607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:39:40.410809 392 external_mini_cluster.cc:1428] Setting key 6598c79b763a2ce3a7478242073ea35b
I20260924 19:39:40.413460 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.405430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60491 (local address 127.0.98.62:37969)
0924 19:39:40.405502 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.405504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.405635 (+ 131us) server_negotiation.cc:407] Connection header received
0924 19:39:40.406091 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.406092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.406102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.406141 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:40.406989 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.407475 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.408459 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.408636 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.408684 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.410735 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:40.410736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:40.410737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:40.410743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:40.412074 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.412509 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.412510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.412510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.412560 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.412811 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.412812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.412813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.412959 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:40.413014 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.413280 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.413376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:39:40.413609 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.405392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:60491)
0924 19:39:40.405517 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.405550 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.406006 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.406203 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.406206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.406724 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:40.406927 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.406931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.407570 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.407571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.408389 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.408392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.408736 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.408738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.408856 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.409272 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:40.409279 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:40.410650 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:40.412156 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.412163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.412165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.412445 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.412621 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.412621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.412622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.412754 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.413012 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:40.413018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:40.413152 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.413233 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.413385 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260924 19:39:40.414065 16121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:39:40.414151 16121 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:40.414312 16121 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:39:40.414655 15796 ts_manager.cc:194] Registered new tserver with Master: 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875)
I20260924 19:39:40.415010 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60491
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:39:40.432178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:39:40.508652 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:39:40.508848 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:39:40.508870 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:39:40.509950 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:39:40.509987 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:39:40.510015 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:39:40.511386 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:39:40.511590 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:39:40.511790 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:39:40.516403 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.516376 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:39:40.516376 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:39:40.516650 16126 server_base.cc:1034] running on GCE node
I20260924 19:39:40.516844 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:39:40.517042 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:39:40.518168 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790278780518156 us; error 29 us; skew 500 ppm
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:39:40.520385 16126 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:39:40.520783 16126 webserver.cc:460] Webserver started at http://127.0.98.3:37927/ using document root <none> and password file <none>
I20260924 19:39:40.521013 16126 fs_manager.cc:359] Metadata directory not provided
I20260924 19:39:40.521056 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:39:40.521174 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:39:40.522121 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "38e0f00131e143cf897cd8ba49b3582f"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "84da579408e5190afa1e6e32645e8566"
server_key_iv: "ff465d5952a69d25961980a4cc34dbe6"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.522432 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38e0f00131e143cf897cd8ba49b3582f"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "84da579408e5190afa1e6e32645e8566"
server_key_iv: "ff465d5952a69d25961980a4cc34dbe6"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.523542 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:39:40.524170 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.524303 16126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:39:40.524380 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "38e0f00131e143cf897cd8ba49b3582f"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "84da579408e5190afa1e6e32645e8566"
server_key_iv: "ff465d5952a69d25961980a4cc34dbe6"
server_key_version: "encryptionkey@0"
I20260924 19:39:40.524439 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:39:40.532590 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:39:40.534090 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:39:40.534207 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:39:40.534385 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:39:40.534649 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:39:40.534685 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.534711 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:39:40.534735 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:39:40.540037 16126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41435
I20260924 19:39:40.540094 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41435 every 8 connection(s)
I20260924 19:39:40.540359 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:39:40 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:39:40.545890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16126
I20260924 19:39:40.545995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:39:40.546209 392 external_mini_cluster.cc:1428] Setting key aef07dbe22cf3320d03444184e74af4c
I20260924 19:39:40.548403 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.540977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41931 (local address 127.0.98.62:37969)
0924 19:39:40.541030 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.541032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.541179 (+ 147us) server_negotiation.cc:407] Connection header received
0924 19:39:40.541610 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.541611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.541620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.541671 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:40.542486 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.542971 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.543855 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.544105 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.544155 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.545930 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:40.545932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:40.545932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:40.545957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:40.547337 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.547657 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.547658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.547658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.547705 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.547883 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.547884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.547884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.548029 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:40.548076 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.548288 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.548341 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:40.548568 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.540935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:41931)
0924 19:39:40.541067 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.541106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.541543 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.541721 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.541725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.542240 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:39:40.542419 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.542430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.543031 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.543031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.543785 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.543786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.544163 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.544163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.544258 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.544629 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:39:40.544635 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:39:40.545851 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:39:40.547401 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.547406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.547408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.547602 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.547750 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:39:40.547751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:39:40.547751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:39:40.547847 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:39:40.548076 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:39:40.548080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:39:40.548186 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.548265 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.548378 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260924 19:39:40.548978 16255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:39:40.549058 16255 heartbeater.cc:464] Registering TS with master...
I20260924 19:39:40.549217 16255 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:39:40.549535 15796 ts_manager.cc:194] Registered new tserver with Master: 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435)
I20260924 19:39:40.549865 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41931
I20260924 19:39:40.557638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:39:40.563864 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.558914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.62:37969)
0924 19:39:40.558978 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.558980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.559057 (+ 77us) server_negotiation.cc:407] Connection header received
0924 19:39:40.559095 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.559096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.559105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.559144 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:39:40.559611 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.560113 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.560768 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.560938 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.560977 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.561925 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:39:40.561926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:39:40.561926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:39:40.561931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:39:40.563058 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.563313 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.563325 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.563325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.563373 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:39:40.563484 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:39:40.563484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:39:40.563484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:39:40.563601 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:39:40.563666 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.563736 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.563790 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:39:40.565164 15796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:39:40.566610 15796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:39:40.572444 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.0.1:41126)
0924 19:39:40.568969 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.568976 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.569037 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.569209 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.569211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.569213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:40.569287 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.569289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.570167 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.570168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.571233 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.571236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.572148 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.572149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.572227 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.572241 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.572315 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.572367 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":95,"threads_started":1}
I20260924 19:39:40.572454 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.1:45099)
0924 19:39:40.568874 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.568879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.569032 (+ 153us) server_negotiation.cc:407] Connection header received
0924 19:39:40.569104 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.569106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.569122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.569192 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:40.569361 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.570079 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.571323 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.572069 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.572139 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.572197 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.572333 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.572372 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260924 19:39:40.572681 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41435 (local address 127.0.0.1:48992)
0924 19:39:40.568674 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.568714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.568792 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.569337 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.569338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.569340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:40.569385 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.569386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.570653 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.570653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.571686 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.571688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.572470 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.572471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.572530 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.572534 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.572563 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.572626 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":11,"thread_start_us":58,"threads_started":1}
I20260924 19:39:40.572944 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.98.3:41435)
0924 19:39:40.569063 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.569067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.569077 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:39:40.569113 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.569115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.569130 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.569453 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:40.569518 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.570556 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.571760 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.572385 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.572737 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.572804 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.572829 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.572865 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260924 19:39:40.573478 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 (DEFAULT_TABLE table=test-table [id=e1fd962d69f74b80b352b09d690b98b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:40.573668 16189 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 (DEFAULT_TABLE table=test-table [id=e1fd962d69f74b80b352b09d690b98b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:40.573925 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7089172ebf4fb4b05bf72db15b8565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:40.573951 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:38322)
0924 19:39:40.569057 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.569067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.569120 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.569340 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.569341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.569342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:40.569385 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.569386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.571776 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.571777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.572896 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.572899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.573737 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.573738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.573796 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.573800 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.573854 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.573891 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":58,"threads_started":1}
I20260924 19:39:40.573978 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7089172ebf4fb4b05bf72db15b8565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:40.574035 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.568803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.98.2:37875)
0924 19:39:40.568972 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.568977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.569117 (+ 140us) server_negotiation.cc:407] Connection header received
0924 19:39:40.569181 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.569183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.569198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.569264 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:40.569457 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.570547 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.572982 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.573661 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.573726 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.573777 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.573924 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.573963 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 19:39:40.574590 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 (DEFAULT_TABLE table=test-table [id=e1fd962d69f74b80b352b09d690b98b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:39:40.574893 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7089172ebf4fb4b05bf72db15b8565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:39:40.576587 16269 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap starting.
I20260924 19:39:40.577018 16270 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Bootstrap starting.
I20260924 19:39:40.577229 16269 tablet_bootstrap.cc:654] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:40.577674 16270 tablet_bootstrap.cc:654] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:40.577783 16271 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Bootstrap starting.
I20260924 19:39:40.577796 16269 log.cc:824] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:40.578259 16270 log.cc:824] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:40.578474 16271 tablet_bootstrap.cc:654] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Neither blocks nor log segments found. Creating new log.
I20260924 19:39:40.578562 16269 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: No bootstrap required, opened a new log
I20260924 19:39:40.578620 16269 ts_tablet_manager.cc:1397] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:40.578924 16271 log.cc:824] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Log is configured to *not* fsync() on all Append() calls
I20260924 19:39:40.579506 16271 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: No bootstrap required, opened a new log
I20260924 19:39:40.579562 16271 ts_tablet_manager.cc:1397] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:39:40.580055 16269 raft_consensus.cc:348] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.580188 16269 raft_consensus.cc:374] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:40.580219 16269 raft_consensus.cc:729] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494854bcfde48deaf02eb8f7bdc111c, State: Initialized, Role: FOLLOWER
I20260924 19:39:40.580339 16269 consensus_queue.cc:260] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.580598 16269 ts_tablet_manager.cc:1428] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:40.580631 15987 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
I20260924 19:39:40.581132 16270 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: No bootstrap required, opened a new log
I20260924 19:39:40.581190 16270 ts_tablet_manager.cc:1397] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:39:40.581311 16271 raft_consensus.cc:348] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.581482 16271 raft_consensus.cc:374] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:40.581527 16271 raft_consensus.cc:729] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6037807dd14115bacd64760efb3e18, State: Initialized, Role: FOLLOWER
I20260924 19:39:40.581633 16271 consensus_queue.cc:260] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.581895 16271 ts_tablet_manager.cc:1428] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:40.581899 16121 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
I20260924 19:39:40.582815 16270 raft_consensus.cc:348] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.582940 16270 raft_consensus.cc:374] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:39:40.582970 16270 raft_consensus.cc:729] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0f00131e143cf897cd8ba49b3582f, State: Initialized, Role: FOLLOWER
I20260924 19:39:40.583066 16270 consensus_queue.cc:260] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.583287 16270 ts_tablet_manager.cc:1428] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:39:40.583329 16255 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
W20260924 19:39:40.655364 16122 tablet.cc:2369] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:39:40.658469 16277 raft_consensus.cc:482] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:39:40.658586 16277 raft_consensus.cc:504] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.659008 16277 leader_election.cc:288] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099), 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875)
I20260924 19:39:40.661553 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.659192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34799 (local address 127.0.98.1:45099)
0924 19:39:40.659235 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.659237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.659245 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:39:40.659331 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.659333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.659346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.659400 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:40.659525 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.660045 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.660898 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.661259 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.661324 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.661334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.661455 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.661490 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:39:40.661553 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.659141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.98.3:34799)
0924 19:39:40.659193 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.659205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.659254 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.659404 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.659405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.659407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:40.659472 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.659474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.660112 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.660113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.660825 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.660827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.661329 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.661330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.661401 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.661404 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.661446 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.661485 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:39:40.661666 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.659333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40841 (local address 127.0.98.2:37875)
0924 19:39:40.659376 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.659378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.659421 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:39:40.659477 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.659478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.659488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.659537 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:39:40.659662 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.660155 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.661029 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.661381 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.661437 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.661446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.661572 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.661607 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:39:40.661672 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.659242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.98.3:40841)
0924 19:39:40.659382 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.659390 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:39:40.659431 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:39:40.659541 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:39:40.659542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:39:40.659543 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:39:40.659607 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.659609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.660220 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.660221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.660954 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:39:40.660956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.661446 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.661446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:39:40.661518 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.661521 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:39:40.661562 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:39:40.661597 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260924 19:39:40.661854 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7089172ebf4fb4b05bf72db15b8565" candidate_uuid: "38e0f00131e143cf897cd8ba49b3582f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c" is_pre_election: true
I20260924 19:39:40.661991 15941 raft_consensus.cc:2454] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e0f00131e143cf897cd8ba49b3582f in term 0.
I20260924 19:39:40.661957 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7089172ebf4fb4b05bf72db15b8565" candidate_uuid: "38e0f00131e143cf897cd8ba49b3582f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6037807dd14115bacd64760efb3e18" is_pre_election: true
I20260924 19:39:40.662088 16075 raft_consensus.cc:2454] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e0f00131e143cf897cd8ba49b3582f in term 0.
I20260924 19:39:40.662194 16143 leader_election.cc:302] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e0f00131e143cf897cd8ba49b3582f, 7494854bcfde48deaf02eb8f7bdc111c; no voters:
I20260924 19:39:40.662341 16277 raft_consensus.cc:2781] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:39:40.662390 16277 raft_consensus.cc:482] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:39:40.662411 16277 raft_consensus.cc:3037] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:40.663050 16277 raft_consensus.cc:504] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.663154 16277 leader_election.cc:288] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [CANDIDATE]: Term 1 election: Requested vote from peers 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099), 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875)
I20260924 19:39:40.663311 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7089172ebf4fb4b05bf72db15b8565" candidate_uuid: "38e0f00131e143cf897cd8ba49b3582f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6037807dd14115bacd64760efb3e18"
I20260924 19:39:40.663324 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7089172ebf4fb4b05bf72db15b8565" candidate_uuid: "38e0f00131e143cf897cd8ba49b3582f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c"
I20260924 19:39:40.663381 16075 raft_consensus.cc:3037] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:40.663383 15941 raft_consensus.cc:3037] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:39:40.664052 16075 raft_consensus.cc:2454] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e0f00131e143cf897cd8ba49b3582f in term 1.
I20260924 19:39:40.664147 15941 raft_consensus.cc:2454] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e0f00131e143cf897cd8ba49b3582f in term 1.
I20260924 19:39:40.664217 16143 leader_election.cc:302] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6037807dd14115bacd64760efb3e18, 38e0f00131e143cf897cd8ba49b3582f; no voters:
I20260924 19:39:40.664325 16277 raft_consensus.cc:2781] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:39:40.664459 16277 raft_consensus.cc:686] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 1 LEADER]: Becoming Leader. State: Replica: 38e0f00131e143cf897cd8ba49b3582f, State: Running, Role: LEADER
I20260924 19:39:40.664538 16277 consensus_queue.cc:237] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:39:40.665405 15795 catalog_manager.cc:5495] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "38e0f00131e143cf897cd8ba49b3582f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } health_report { overall_health: UNKNOWN } } }
I20260924 19:39:40.685859 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.683674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.0.98.3:41435)
0924 19:39:40.683730 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:39:40.683732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:39:40.683750 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:39:40.683786 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:39:40.683787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:39:40.683806 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:39:40.683864 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:39:40.684007 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.684468 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.685188 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:39:40.685381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:39:40.685434 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:39:40.685574 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:39:40.685674 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:39:40.685743 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:39:40.685763 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 19:39:40.685783 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:39:40.687500 15941 raft_consensus.cc:1261] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Refusing update from remote peer 38e0f00131e143cf897cd8ba49b3582f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:40.687505 16075 raft_consensus.cc:1261] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Refusing update from remote peer 38e0f00131e143cf897cd8ba49b3582f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:39:40.687801 16279 consensus_queue.cc:1035] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:40.687925 16277 consensus_queue.cc:1035] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:39:40.691241 16284 mvcc.cc:205] Tried to move back new op lower bound from 7332981885693636608 to 7332981885602586624. Current Snapshot: MvccSnapshot[applied={T|T < 7332981885693636608 or (T in {7332981885693636608})}]
I20260924 19:39:40.692040 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:40.692117 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:39:40.692807 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Sending DeleteTablet for 3 replicas of tablet 5c7089172ebf4fb4b05bf72db15b8565
I20260924 19:39:40.693092 16189 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48992
I20260924 19:39:40.693135 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41126
I20260924 19:39:40.693202 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38322
I20260924 19:39:40.693356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15858
I20260924 19:39:40.693931 16292 tablet_replica.cc:330] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: stopping tablet replica
I20260924 19:39:40.694046 16292 raft_consensus.cc:2229] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:39:40.694110 16294 tablet_replica.cc:330] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: stopping tablet replica
I20260924 19:39:40.694199 16294 raft_consensus.cc:2229] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:39:40.694170 16292 raft_consensus.cc:2258] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:40.694303 16294 raft_consensus.cc:2258] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:39:40.694700 16292 ts_tablet_manager.cc:1905] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:40.694749 16294 ts_tablet_manager.cc:1905] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:39:40.696022 16292 ts_tablet_manager.cc:1918] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:40.696087 16292 log.cc:1196] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/5c7089172ebf4fb4b05bf72db15b8565
I20260924 19:39:40.696293 16294 ts_tablet_manager.cc:1918] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:39:40.696360 16294 log.cc:1196] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/5c7089172ebf4fb4b05bf72db15b8565
I20260924 19:39:40.696393 16292 ts_tablet_manager.cc:1939] T 5c7089172ebf4fb4b05bf72db15b8565 P 38e0f00131e143cf897cd8ba49b3582f: Deleting consensus metadata
I20260924 19:39:40.696599 16294 ts_tablet_manager.cc:1939] T 5c7089172ebf4fb4b05bf72db15b8565 P 2e6037807dd14115bacd64760efb3e18: Deleting consensus metadata
I20260924 19:39:40.696740 15780 catalog_manager.cc:4841] TS 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435): tablet 5c7089172ebf4fb4b05bf72db15b8565 (table test-table [id=e1fd962d69f74b80b352b09d690b98b1]) successfully deleted
I20260924 19:39:40.696943 15781 catalog_manager.cc:4841] TS 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875): tablet 5c7089172ebf4fb4b05bf72db15b8565 (table test-table [id=e1fd962d69f74b80b352b09d690b98b1]) successfully deleted
W20260924 19:39:40.698796 15781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45099 (error 108)
I20260924 19:39:40.699010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15992
I20260924 19:39:40.699213 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:39:40.699105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.0.1:41142)
0924 19:39:40.699153 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:39:40.699183 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:39:40.699329 15781 catalog_manager.cc:4584] TS 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c7089172ebf4fb4b05bf72db15b8565: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45099: connect: Connection refused (error 111)
I20260924 19:39:40.703245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16126
I20260924 19:39:40.707799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 15764
I20260924 19:40:00.712167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--webserver_interface=127.0.98.62
--webserver_port=37553
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:00.790536 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:00.790726 16296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:00.790747 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:00.791882 16296 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:40:00.791920 16296 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:40:00.791950 16296 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:40:00.791963 16296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:00.791987 16296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:00.792003 16296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:00.792025 16296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:00.792040 16296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:00.793470 16296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:00.793687 16296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:00.793896 16296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:00.801723 16303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:00.801891 16296 server_base.cc:1034] running on GCE node
W20260924 19:40:00.801785 16305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:00.801680 16302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:00.802217 16296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:00.802534 16296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:00.803673 16296 hybrid_clock.cc:648] HybridClock initialized: now 1790278800803663 us; error 26 us; skew 500 ppm
Sep 24 19:40:00 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:00.814272 16296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:40:00.814720 16296 webserver.cc:460] Webserver started at http://127.0.98.62:37553/ using document root <none> and password file <none>
I20260924 19:40:00.814908 16296 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:00.814957 16296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:00.816298 16296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:00.816877 16312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:00.817025 16296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:00.817104 16296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "2b445e07a610482a92096698aa40eb2a"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "cd26356fe5a90d12a9a206f372a6c17e"
server_key_iv: "a6fc0bb8c8fab7576733395c37815ed0"
server_key_version: "encryptionkey@0"
I20260924 19:40:00.817493 16296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:00.837368 16296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:00.839399 16296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:00.839504 16296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:00.843150 16296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37969
I20260924 19:40:00.843186 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37969 every 8 connection(s)
I20260924 19:40:00.843533 16296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:00.844954 16365 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:40:00.845594 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap starting.
I20260924 19:40:00.847057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16296
I20260924 19:40:00.847337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45099
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=39285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:40:00.853938 16365 log.cc:824] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:00.855737 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:00.856038 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap complete.
I20260924 19:40:00.858201 16365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:00.858430 16365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Initialized, Role: FOLLOWER
I20260924 19:40:00.858595 16365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:00.858654 16365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:00.858703 16365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:00.858750 16365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:40:00.859753 16365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:00.859843 16365 leader_election.cc:302] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b445e07a610482a92096698aa40eb2a; no voters:
I20260924 19:40:00.860018 16365 leader_election.cc:288] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:40:00.860061 16369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:40:00.860221 16369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 LEADER]: Becoming Leader. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Running, Role: LEADER
I20260924 19:40:00.860272 16365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:00.860417 16369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:00.861074 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b445e07a610482a92096698aa40eb2a. Latest consensus state: current_term: 2 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:40:00.861064 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:40:00.861152 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:00.861158 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:00.861318 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:00.862043 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:00.862135 16379 catalog_manager.cc:1259] Loaded cluster ID: bd84d08babb14de497b09735c3e514df
I20260924 19:40:00.862200 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:00.863351 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:00.863529 16379 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Loaded TSK: 0
I20260924 19:40:00.876365 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Generated new TSK 1
I20260924 19:40:00.876468 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:40:00.928221 16367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:00.928428 16367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:00.928462 16367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:00.929572 16367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:00.929610 16367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:00.929643 16367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:00.931038 16367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39285
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:00.931262 16367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:00.931459 16367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:00.936276 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:00.936254 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:00.936398 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:00.936456 16367 server_base.cc:1034] running on GCE node
I20260924 19:40:00.936790 16367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:00.937084 16367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:00.938265 16367 hybrid_clock.cc:648] HybridClock initialized: now 1790278800938239 us; error 39 us; skew 500 ppm
Sep 24 19:40:00 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:00.941187 16367 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:40:00.941732 16367 webserver.cc:460] Webserver started at http://127.0.98.1:39285/ using document root <none> and password file <none>
I20260924 19:40:00.942008 16367 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:00.942090 16367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:00.943828 16367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:40:00.944572 16402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:00.944741 16367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:00.944828 16367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "7494854bcfde48deaf02eb8f7bdc111c"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "0a8acfeeb25a078b238907f7f7ec0b5f"
server_key_iv: "934d214008e01014ff4a08740325a166"
server_key_version: "encryptionkey@0"
I20260924 19:40:00.945147 16367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:00.962684 16367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:00.964753 16367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:00.964866 16367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:00.965082 16367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:00.965502 16409 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:40:00.967316 16367 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:40:00.967373 16367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:40:00.967411 16367 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:40:00.968087 16367 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:40:00.968120 16367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:00.968201 16409 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap starting.
I20260924 19:40:00.974611 16367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45099
I20260924 19:40:00.974663 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45099 every 8 connection(s)
I20260924 19:40:00.975031 16367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:00.979373 16409 log.cc:824] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:00.982342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16367
I20260924 19:40:00.982398 16409 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:00.982622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37875
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=35815
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:40:00.982673 16409 tablet_bootstrap.cc:492] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap complete.
I20260924 19:40:00.983721 16409 ts_tablet_manager.cc:1397] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260924 19:40:00.985021 16409 raft_consensus.cc:348] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:40:00.985347 16409 raft_consensus.cc:729] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494854bcfde48deaf02eb8f7bdc111c, State: Initialized, Role: FOLLOWER
I20260924 19:40:00.985530 16409 consensus_queue.cc:260] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } } peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260924 19:40:00.985781 16409 ts_tablet_manager.cc:1428] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 24 19:40:00 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:00.990178 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:00.975884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42923 (local address 127.0.98.62:37969)
0924 19:40:00.979878 (+ 3994us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:00.979888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:00.979953 (+ 65us) server_negotiation.cc:407] Connection header received
0924 19:40:00.980089 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:00.980096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:00.980151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:00.980286 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:00.980928 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.981843 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.983190 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.983769 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.984368 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:00.986354 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:00.986358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:00.986358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:00.986370 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:00.988296 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:00.988742 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:00.988745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:00.988746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:00.988856 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:00.989137 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:00.989138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:00.989138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:00.989415 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:00.989500 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:00.989847 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:40:00.989990 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3911,"thread_start_us":92,"threads_started":1}
I20260924 19:40:00.990190 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:00.975805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:42923)
0924 19:40:00.976038 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:00.976082 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:00.976550 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:00.980344 (+ 3794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:00.980348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:00.980599 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:00.980798 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:00.980802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.981962 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.981963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:00.983095 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:00.983098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.983890 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.983892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:00.984063 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:00.984627 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:00.984637 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:00.986240 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:00.988378 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:00.988385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:00.988388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:00.988661 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:00.988935 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:00.988936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:00.988936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:00.989075 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:00.989489 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:00.989496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:00.989666 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 19:40:00.989786 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 19:40:00.989965 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":125,"threads_started":1}
I20260924 19:40:00.991225 16517 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:40:00.991333 16517 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:00.991537 16517 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:40:00.992324 16329 ts_manager.cc:194] Registered new tserver with Master: 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
I20260924 19:40:00.993482 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42923
I20260924 19:40:00.994481 16517 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
I20260924 19:40:00.998390 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:00.993743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.0.98.1:45099)
0924 19:40:00.994117 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:00.994121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:00.994130 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:00.994158 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:00.994160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:00.994327 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:00.994403 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:00.994596 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.995807 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.997300 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.998022 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.998094 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:00.998154 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:00.998273 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:40:00.998314 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":80,"threads_started":1}
I20260924 19:40:00.998399 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:00.992920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.0.1:58778)
0924 19:40:00.993914 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:00.993947 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:00.994008 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:00.994465 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:00.994468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:00.994469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:00.994528 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:00.994529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.995893 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.995895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:00.997210 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:00.997213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:00.998099 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:00.998100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:00.998184 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:00.998198 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:40:00.998261 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:40:00.998322 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":89,"threads_started":1}
I20260924 19:40:00.998799 16451 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7089172ebf4fb4b05bf72db15b8565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58778
I20260924 19:40:00.999466 16526 tablet_replica.cc:330] stopping tablet replica
I20260924 19:40:00.999569 16526 raft_consensus.cc:2229] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:00.999639 16526 raft_consensus.cc:2258] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:00.999909 16526 ts_tablet_manager.cc:1905] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:01.001981 16526 ts_tablet_manager.cc:1918] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:01.002058 16526 log.cc:1196] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal/wals/5c7089172ebf4fb4b05bf72db15b8565
I20260924 19:40:01.002233 16526 ts_tablet_manager.cc:1939] T 5c7089172ebf4fb4b05bf72db15b8565 P 7494854bcfde48deaf02eb8f7bdc111c: Deleting consensus metadata
I20260924 19:40:01.002738 16314 catalog_manager.cc:4841] TS 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099): tablet 5c7089172ebf4fb4b05bf72db15b8565 (table test-table [id=e1fd962d69f74b80b352b09d690b98b1]) successfully deleted
W20260924 19:40:01.067353 16522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:01.067551 16522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:01.067574 16522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:01.068678 16522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:01.068713 16522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:01.068737 16522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:01.070139 16522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35815
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:01.070322 16522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:01.070533 16522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:01.075084 16531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:01.075230 16522 server_base.cc:1034] running on GCE node
W20260924 19:40:01.075150 16532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:01.075101 16534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:01.075539 16522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:01.075778 16522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:01.076938 16522 hybrid_clock.cc:648] HybridClock initialized: now 1790278801076912 us; error 34 us; skew 500 ppm
Sep 24 19:40:01 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:01.079267 16522 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:40:01.079689 16522 webserver.cc:460] Webserver started at http://127.0.98.2:35815/ using document root <none> and password file <none>
I20260924 19:40:01.079995 16522 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:01.080089 16522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:01.081446 16522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:01.081941 16541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.082100 16522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:01.082177 16522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "2e6037807dd14115bacd64760efb3e18"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "4fb2edb15c1006c98d6da8682d148971"
server_key_iv: "9fe69436a7794e71a0641ec6ded2e407"
server_key_version: "encryptionkey@0"
I20260924 19:40:01.082505 16522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:01.092046 16522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:01.093858 16522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:01.093982 16522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:01.094168 16522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:01.094444 16522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:01.094475 16522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.094502 16522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:01.094525 16522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.100013 16522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260924 19:40:01.100056 16654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260924 19:40:01.100361 16522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:01.100889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16522
I20260924 19:40:01.101181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41435
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=37927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:40:01 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:01.109305 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.101041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44241 (local address 127.0.98.62:37969)
0924 19:40:01.101083 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.101085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.101350 (+ 265us) server_negotiation.cc:407] Connection header received
0924 19:40:01.101871 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.101873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.101885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.101951 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:01.102848 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.103354 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.104374 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.104590 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.104646 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.106813 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:01.106815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:01.106815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:01.106822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:01.108099 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.108451 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.108452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.108452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.108503 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.108708 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.108709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.108709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.108814 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:01.108868 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.109124 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.109206 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:40:01.109496 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.101056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:44241)
0924 19:40:01.101216 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.101262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.101768 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.102004 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.102007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.102562 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:01.102747 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.102751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.103435 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.103436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.104295 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.104298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.104659 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.104660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.104775 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.105235 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:01.105243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:01.106724 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:01.108167 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:01.108174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:01.108176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:01.108387 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:01.108556 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:01.108557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:01.108557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:01.108655 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:01.108866 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:01.108873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:01.109010 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.109095 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.109245 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260924 19:40:01.110009 16655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:40:01.110111 16655 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:01.110291 16655 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:40:01.110648 16329 ts_manager.cc:194] Registered new tserver with Master: 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875)
I20260924 19:40:01.111155 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44241
W20260924 19:40:01.181718 16658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:01.181911 16658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:01.181936 16658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:01.183025 16658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:01.183060 16658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:01.183084 16658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:01.184474 16658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37927
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:01.184674 16658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:01.184901 16658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:01.189723 16663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:01.189853 16658 server_base.cc:1034] running on GCE node
W20260924 19:40:01.189783 16664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:01.189941 16666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:01.190138 16658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:01.190373 16658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:01.191514 16658 hybrid_clock.cc:648] HybridClock initialized: now 1790278801191502 us; error 26 us; skew 500 ppm
Sep 24 19:40:01 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:01.193874 16658 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:40:01.194283 16658 webserver.cc:460] Webserver started at http://127.0.98.3:37927/ using document root <none> and password file <none>
I20260924 19:40:01.194496 16658 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:01.194553 16658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:01.195855 16658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:01.196586 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.196749 16658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.196826 16658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "38e0f00131e143cf897cd8ba49b3582f"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "84da579408e5190afa1e6e32645e8566"
server_key_iv: "ff465d5952a69d25961980a4cc34dbe6"
server_key_version: "encryptionkey@0"
I20260924 19:40:01.197155 16658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:01.221607 16658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:01.223359 16658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:01.223497 16658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:01.223719 16658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:01.224035 16658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:01.224071 16658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.224099 16658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:01.224119 16658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:01.229552 16658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41435
I20260924 19:40:01.229631 16786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41435 every 8 connection(s)
I20260924 19:40:01.229930 16658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:40:01 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:01.235983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16658
I20260924 19:40:01.238492 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.230602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54189 (local address 127.0.98.62:37969)
0924 19:40:01.230666 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.230668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.230876 (+ 208us) server_negotiation.cc:407] Connection header received
0924 19:40:01.231308 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.231309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.231319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.231383 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:01.232344 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.232830 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.233746 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.233931 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.233985 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.235912 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:01.235913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:01.235913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:01.235920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:01.237212 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.237624 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.237624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.237624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.237669 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.237900 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.237900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.237901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.237987 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:01.238039 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.238302 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.238391 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:40:01.238649 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.230580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:54189)
0924 19:40:01.230729 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.230769 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.231234 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.231441 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.231445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.232020 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:01.232260 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.232266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.232888 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.232889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.233677 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.233679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.233987 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.233988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.234077 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.234533 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:01.234541 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:01.235807 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:01.237276 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:01.237283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:01.237285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:01.237561 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:01.237718 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:01.237718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:01.237719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:01.237838 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:01.238051 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:01.238058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:01.238188 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.238274 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.238423 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260924 19:40:01.239140 16787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37969
I20260924 19:40:01.239241 16787 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:01.239435 16787 heartbeater.cc:511] Master 127.0.98.62:37969 requested a full tablet report, sending...
I20260924 19:40:01.239814 16329 ts_manager.cc:194] Registered new tserver with Master: 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435)
I20260924 19:40:01.240185 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54189
I20260924 19:40:01.242403 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.236324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.98.62:37969)
0924 19:40:01.236431 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.236433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.236526 (+ 93us) server_negotiation.cc:407] Connection header received
0924 19:40:01.236589 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.236590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.236601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.236652 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:01.237196 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.237902 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.238663 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.238935 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.238995 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.240246 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:01.240247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:01.240247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:01.240253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:01.241481 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.241864 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.241865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.241865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.241908 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.242039 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.242039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.242040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.242120 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:01.242237 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.242266 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.242321 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
I20260924 19:40:01.242990 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:01.245468 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.243295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.0.98.62:37969)
0924 19:40:01.243329 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.243330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.243453 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:40:01.243497 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.243498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.243526 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.243578 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:01.243649 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.244187 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.244942 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.245122 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.245167 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.245270 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:01.245308 (+ 38us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:40:01.245421 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54452: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:40:01.245520 16790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54452: token expired
W20260924 19:40:01.245626 16261 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37969)
I20260924 19:40:01.250553 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.245862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.0.98.62:37969)
0924 19:40:01.245907 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.245908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.245936 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:40:01.245980 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.245980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.245987 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.246026 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:01.246398 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.246806 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.247454 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.247610 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.247649 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.248637 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:01.248638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:01.248638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:01.248644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:01.249745 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.250041 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.250041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.250042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.250085 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:01.250211 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:01.250212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:01.250212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:01.250313 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:01.250381 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.250446 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.250498 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:40:01.251713 16329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:01.253337 16329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:01.255276 16451 tablet_service.cc:1467] Processing CreateTablet for tablet 51dacaf7ad2043eb876c3532190532bf (DEFAULT_TABLE table=test-table [id=1bc326970ca04f9ea689a0c56c411d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:01.255483 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dacaf7ad2043eb876c3532190532bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:01.256995 16409 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap starting.
I20260924 19:40:01.257366 16409 tablet_bootstrap.cc:654] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:01.257956 16409 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: No bootstrap required, opened a new log
I20260924 19:40:01.258010 16409 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:01.258155 16409 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.258234 16409 raft_consensus.cc:374] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:01.258281 16409 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494854bcfde48deaf02eb8f7bdc111c, State: Initialized, Role: FOLLOWER
I20260924 19:40:01.258322 16409 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.258411 16409 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:01.258700 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.254966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:41452)
0924 19:40:01.255016 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.255027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.255058 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.255444 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.255445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.255447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:01.255488 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.255489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.256517 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.256518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.257606 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.257609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.258496 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.258497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.258561 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.258564 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.258593 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.258633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:40:01.259057 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.255066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.0.98.2:37875)
0924 19:40:01.255270 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.255275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.255287 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:40:01.255327 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.255329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.255346 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.255430 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:01.255529 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.256452 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.257676 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.258384 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.258846 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.258914 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.258939 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.258978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260924 19:40:01.259173 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.255021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41435 (local address 127.0.0.1:36822)
0924 19:40:01.255159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.255168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.255202 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.255570 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.255571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.255572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:01.255599 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.255601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.256747 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.256748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.257764 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.257766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.258919 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.258920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.258987 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.258990 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.259040 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.259101 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260924 19:40:01.259403 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.255248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.98.3:41435)
0924 19:40:01.255425 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.255430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.255445 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:40:01.255481 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.255483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.255497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.255575 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:01.255654 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.256666 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.257855 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.258643 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.258729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.258794 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.259284 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.259327 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260924 19:40:01.259665 16589 tablet_service.cc:1467] Processing CreateTablet for tablet 51dacaf7ad2043eb876c3532190532bf (DEFAULT_TABLE table=test-table [id=1bc326970ca04f9ea689a0c56c411d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:01.260007 16589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dacaf7ad2043eb876c3532190532bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:01.260061 16721 tablet_service.cc:1467] Processing CreateTablet for tablet 51dacaf7ad2043eb876c3532190532bf (DEFAULT_TABLE table=test-table [id=1bc326970ca04f9ea689a0c56c411d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:01.260354 16721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dacaf7ad2043eb876c3532190532bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:01.262719 16797 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Bootstrap starting.
I20260924 19:40:01.262989 16798 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Bootstrap starting.
I20260924 19:40:01.263324 16797 tablet_bootstrap.cc:654] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:01.263538 16798 tablet_bootstrap.cc:654] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:01.263906 16797 log.cc:824] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:01.264060 16798 log.cc:824] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:01.264721 16798 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: No bootstrap required, opened a new log
I20260924 19:40:01.264721 16797 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: No bootstrap required, opened a new log
I20260924 19:40:01.264788 16797 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:40:01.264788 16798 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:01.266590 16798 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.266590 16797 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.266758 16798 raft_consensus.cc:374] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:01.266758 16797 raft_consensus.cc:374] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:01.266800 16797 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6037807dd14115bacd64760efb3e18, State: Initialized, Role: FOLLOWER
I20260924 19:40:01.266800 16798 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0f00131e143cf897cd8ba49b3582f, State: Initialized, Role: FOLLOWER
I20260924 19:40:01.266919 16797 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.266919 16798 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.267185 16797 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:40:01.267213 16655 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
I20260924 19:40:01.267393 16787 heartbeater.cc:503] Master 127.0.98.62:37969 was elected leader, sending a full tablet report...
I20260924 19:40:01.267428 16798 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 19:40:01.350867 16656 tablet.cc:2369] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:01.460656 16801 raft_consensus.cc:482] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:01.460785 16801 raft_consensus.cc:504] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.461089 16801 leader_election.cc:288] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435), 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
I20260924 19:40:01.464963 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.461313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.98.2:42835)
0924 19:40:01.461376 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.461387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.461432 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.461683 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.461684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.461685 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:01.461758 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.461760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.462616 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.462616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.463896 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.463899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.464664 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.464768 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.464774 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.464827 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.464873 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":70}
I20260924 19:40:01.464983 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.461509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42835 (local address 127.0.98.1:45099)
0924 19:40:01.461569 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.461571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.461581 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:40:01.461610 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.461611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.461622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.461678 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:01.461814 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.462545 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464006 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.464576 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464663 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.464672 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.464869 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.464913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:40:01.464983 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.461332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56963 (local address 127.0.98.3:41435)
0924 19:40:01.461384 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.461386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.461626 (+ 240us) server_negotiation.cc:407] Connection header received
0924 19:40:01.461652 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.461653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.461666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.461709 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:01.461818 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.462561 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464007 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.464597 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464661 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.464672 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.464869 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.464913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:01.464970 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.461319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41435 (local address 127.0.98.2:56963)
0924 19:40:01.461529 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.461537 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:01.461582 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:01.461714 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:01.461715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:01.461716 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:01.461764 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.461765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.462617 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.462618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.463896 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:01.463899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.464664 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.464664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:01.464768 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.464775 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:40:01.464827 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:40:01.464888 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":105,"thread_start_us":92,"threads_started":1}
I20260924 19:40:01.465315 16471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c" is_pre_election: true
I20260924 19:40:01.465517 16471 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 0.
I20260924 19:40:01.465544 16741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0f00131e143cf897cd8ba49b3582f" is_pre_election: true
I20260924 19:40:01.465709 16741 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 0.
I20260924 19:40:01.465777 16543 leader_election.cc:302] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6037807dd14115bacd64760efb3e18, 7494854bcfde48deaf02eb8f7bdc111c; no voters:
I20260924 19:40:01.465902 16801 raft_consensus.cc:2781] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:01.465967 16801 raft_consensus.cc:482] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:01.465991 16801 raft_consensus.cc:3037] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:01.466642 16801 raft_consensus.cc:504] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.466764 16801 leader_election.cc:288] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 1 election: Requested vote from peers 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435), 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
I20260924 19:40:01.466902 16471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c"
I20260924 19:40:01.466931 16741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0f00131e143cf897cd8ba49b3582f"
I20260924 19:40:01.466975 16471 raft_consensus.cc:3037] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:01.466997 16741 raft_consensus.cc:3037] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:01.467646 16741 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 1.
I20260924 19:40:01.467810 16542 leader_election.cc:302] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6037807dd14115bacd64760efb3e18, 38e0f00131e143cf897cd8ba49b3582f; no voters:
I20260924 19:40:01.467896 16801 raft_consensus.cc:2781] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:01.468027 16471 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 1.
I20260924 19:40:01.468039 16801 raft_consensus.cc:686] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6037807dd14115bacd64760efb3e18, State: Running, Role: LEADER
I20260924 19:40:01.468128 16801 consensus_queue.cc:237] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:01.468902 16329 catalog_manager.cc:5495] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6037807dd14115bacd64760efb3e18 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2e6037807dd14115bacd64760efb3e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } health_report { overall_health: UNKNOWN } } }
W20260924 19:40:01.480406 16788 tablet.cc:2369] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:01.482333 16518 tablet.cc:2369] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:01.484297 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.482180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.98.2:37875)
0924 19:40:01.482232 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:01.482235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:01.482245 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:40:01.482283 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:01.482283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:01.482304 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:01.482377 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:01.482496 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.482990 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.483649 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:01.483858 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:01.483908 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:01.484055 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:01.484145 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:01.484200 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:01.484213 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 19:40:01.484229 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:40:01.485853 16741 raft_consensus.cc:1261] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Refusing update from remote peer 2e6037807dd14115bacd64760efb3e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:01.485852 16471 raft_consensus.cc:1261] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Refusing update from remote peer 2e6037807dd14115bacd64760efb3e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:01.486152 16804 consensus_queue.cc:1035] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:01.486264 16801 consensus_queue.cc:1035] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:01.488529 16809 mvcc.cc:205] Tried to move back new op lower bound from 7332981970883653632 to 7332981970813898752. Current Snapshot: MvccSnapshot[applied={T|T < 7332981970883653632}]
I20260924 19:40:01.490424 16329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:01.490515 16329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:01.491144 16329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Sending DeleteTablet for 3 replicas of tablet 51dacaf7ad2043eb876c3532190532bf
I20260924 19:40:01.491429 16721 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dacaf7ad2043eb876c3532190532bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36822
I20260924 19:40:01.491451 16589 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dacaf7ad2043eb876c3532190532bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41452
I20260924 19:40:01.491475 16451 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dacaf7ad2043eb876c3532190532bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58778
I20260924 19:40:01.491544 16526 tablet_replica.cc:330] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: stopping tablet replica
I20260924 19:40:01.491582 16526 raft_consensus.cc:2229] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:01.491619 16526 raft_consensus.cc:2258] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:01.491793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16367
I20260924 19:40:01.496177 16817 tablet_replica.cc:330] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: stopping tablet replica
I20260924 19:40:01.496289 16817 raft_consensus.cc:2229] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:01.496380 16817 raft_consensus.cc:2258] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:01.496483 16818 tablet_replica.cc:330] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: stopping tablet replica
I20260924 19:40:01.496618 16818 raft_consensus.cc:2229] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:01.496673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16522
W20260924 19:40:01.496682 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45099 (error 108)
I20260924 19:40:01.496797 16818 raft_consensus.cc:2258] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:01.497874 16818 ts_tablet_manager.cc:1905] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:01.498317 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.498177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.0.1:58784)
0924 19:40:01.498239 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.498284 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 19:40:01.498440 16314 catalog_manager.cc:4584] TS 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51dacaf7ad2043eb876c3532190532bf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45099: connect: Connection refused (error 111)
I20260924 19:40:01.499383 16818 ts_tablet_manager.cc:1918] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:01.499437 16818 log.cc:1196] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/51dacaf7ad2043eb876c3532190532bf
I20260924 19:40:01.499665 16818 ts_tablet_manager.cc:1939] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Deleting consensus metadata
I20260924 19:40:01.500054 16313 catalog_manager.cc:4841] TS 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435): tablet 51dacaf7ad2043eb876c3532190532bf (table test-table [id=1bc326970ca04f9ea689a0c56c411d0b]) successfully deleted
I20260924 19:40:01.501279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16658
I20260924 19:40:01.501322 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:01.501223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:41476)
0924 19:40:01.501263 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:01.501287 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:40:01.505313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16296
I20260924 19:40:21.509680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--webserver_interface=127.0.98.62
--webserver_port=37553
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:21.588675 16820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:21.588907 16820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:21.588941 16820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:21.590133 16820 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:40:21.590175 16820 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:40:21.590193 16820 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:40:21.590207 16820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:21.590222 16820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:21.590240 16820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:21.590258 16820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:21.590282 16820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:21.591897 16820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:21.592185 16820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:21.592394 16820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:21.597453 16828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:21.597472 16820 server_base.cc:1034] running on GCE node
W20260924 19:40:21.597448 16826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:21.597486 16825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:21.597965 16820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:21.598246 16820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:21.599400 16820 hybrid_clock.cc:648] HybridClock initialized: now 1790278821599392 us; error 28 us; skew 500 ppm
Sep 24 19:40:21 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:21.605543 16820 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:40:21.606053 16820 webserver.cc:460] Webserver started at http://127.0.98.62:37553/ using document root <none> and password file <none>
I20260924 19:40:21.606295 16820 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:21.606344 16820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:21.607685 16820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:40:21.608371 16835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:21.608539 16820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:21.608623 16820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "2b445e07a610482a92096698aa40eb2a"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "cd26356fe5a90d12a9a206f372a6c17e"
server_key_iv: "a6fc0bb8c8fab7576733395c37815ed0"
server_key_version: "encryptionkey@0"
I20260924 19:40:21.608968 16820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:21.619863 16820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:21.621987 16820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:21.622116 16820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:21.625962 16820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37969
I20260924 19:40:21.626000 16887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37969 every 8 connection(s)
I20260924 19:40:21.626354 16820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:21.627754 16888 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:40:21.628420 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap starting.
I20260924 19:40:21.634593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16820
I20260924 19:40:21.634913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45099
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=39285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:40:21.636792 16888 log.cc:824] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:21.639415 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:21.639712 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Bootstrap complete.
I20260924 19:40:21.641826 16888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:21.642059 16888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Initialized, Role: FOLLOWER
I20260924 19:40:21.642207 16888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:21.642274 16888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:21.642304 16888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:21.642354 16888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:40:21.643364 16888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:21.643456 16888 leader_election.cc:302] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b445e07a610482a92096698aa40eb2a; no voters:
I20260924 19:40:21.643632 16888 leader_election.cc:288] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:40:21.643685 16892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:40:21.643853 16892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [term 3 LEADER]: Becoming Leader. State: Replica: 2b445e07a610482a92096698aa40eb2a, State: Running, Role: LEADER
I20260924 19:40:21.643924 16888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:21.644027 16892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } }
I20260924 19:40:21.644343 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b445e07a610482a92096698aa40eb2a. Latest consensus state: current_term: 3 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:40:21.644588 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:21.644515 16894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b445e07a610482a92096698aa40eb2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b445e07a610482a92096698aa40eb2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37969 } } }
I20260924 19:40:21.644742 16894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:21.645100 16901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:21.645877 16901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:21.645965 16901 catalog_manager.cc:1259] Loaded cluster ID: bd84d08babb14de497b09735c3e514df
I20260924 19:40:21.645998 16901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:21.646783 16901 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:21.646943 16901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Loaded TSK: 0
I20260924 19:40:21.646982 16901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b445e07a610482a92096698aa40eb2a: Loaded TSK: 1
F20260924 19:40:21.647171 16901 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fac6f41a9ad google::LogMessage::Fail() at ??:0
@ 0x7fac6f41d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fac6f41a4de google::LogMessage::Flush() at ??:0
@ 0x7fac6f41ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fac71b87104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fac71b84d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fac773741cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fac7737448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fac77374594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fac7733ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fac77344ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fac712e80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fac712dd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fac721416db start_thread at ??:0
@ 0x7fac6df1861f clone at ??:0
W20260924 19:40:21.749106 16890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:21.749439 16890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:21.749509 16890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:21.750674 16890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:21.750720 16890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:21.750754 16890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:21.752223 16890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39285
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:21.752441 16890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:21.752643 16890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:21.757431 16930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:21.757392 16929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:21.757685 16890 server_base.cc:1034] running on GCE node
W20260924 19:40:21.757402 16932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:21.757956 16890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:21.758219 16890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:21.759361 16890 hybrid_clock.cc:648] HybridClock initialized: now 1790278821759341 us; error 34 us; skew 500 ppm
Sep 24 19:40:21 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:21.761705 16890 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:40:21.762111 16890 webserver.cc:460] Webserver started at http://127.0.98.1:39285/ using document root <none> and password file <none>
I20260924 19:40:21.762317 16890 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:21.762362 16890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:21.763648 16890 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:21.764285 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:21.764439 16890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:21.764513 16890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "7494854bcfde48deaf02eb8f7bdc111c"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "0a8acfeeb25a078b238907f7f7ec0b5f"
server_key_iv: "934d214008e01014ff4a08740325a166"
server_key_version: "encryptionkey@0"
I20260924 19:40:21.764842 16890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:21.801903 16890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:21.804039 16890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:21.804183 16890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:21.804410 16890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:21.804832 16946 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:40:21.806864 16890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:40:21.806913 16890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:40:21.806946 16890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:40:21.807559 16890 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:40:21.807593 16890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:40:21.807688 16946 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap starting.
I20260924 19:40:21.809538 16946 log.cc:824] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:21.812587 16946 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:21.812892 16946 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Bootstrap complete.
I20260924 19:40:21.813057 16946 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 19:40:21.814018 16890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45099
I20260924 19:40:21.814301 16890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:21.814661 17054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45099 every 8 connection(s)
I20260924 19:40:21.814882 16946 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:21.815212 16946 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494854bcfde48deaf02eb8f7bdc111c, State: Initialized, Role: FOLLOWER
I20260924 19:40:21.815344 16946 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:21.815589 16946 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:40:21.820583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 16890
I20260924 19:40:21.820842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37875
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=35815
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:21.865679 17057 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:40:21.814981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:44521)
0924 19:40:21.815134 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.815182 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:21.815323 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:21.865461 (+ 50138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260924 19:40:21.865954 17055 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37969 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37969: Client connection negotiation failed: client connection to 127.0.98.62:37969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:40:21.866294 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.866174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:49075)
0924 19:40:21.866223 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.866269 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:40:21.866618 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.866540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:39543)
0924 19:40:21.866578 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.866602 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:40:21.866725 17055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:40:21.902302 17059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:21.902513 17059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:21.902536 17059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:21.903672 17059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:21.903708 17059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:21.903733 17059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:21.905189 17059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35815
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:21.905375 17059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:21.905561 17059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:40:21.910341 17059 server_base.cc:1034] running on GCE node
W20260924 19:40:21.910310 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:21.910370 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:21.910298 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:21.910717 17059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:21.910925 17059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:21.912046 17059 hybrid_clock.cc:648] HybridClock initialized: now 1790278821912037 us; error 25 us; skew 500 ppm
Sep 24 19:40:21 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:21.914343 17059 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:40:21.914729 17059 webserver.cc:460] Webserver started at http://127.0.98.2:35815/ using document root <none> and password file <none>
I20260924 19:40:21.914920 17059 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:21.914974 17059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:21.916347 17059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:21.916934 17074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:21.917112 17059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:21.917191 17059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "2e6037807dd14115bacd64760efb3e18"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "4fb2edb15c1006c98d6da8682d148971"
server_key_iv: "9fe69436a7794e71a0641ec6ded2e407"
server_key_version: "encryptionkey@0"
I20260924 19:40:21.917520 17059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:21.924917 17059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:21.926955 17059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:21.927090 17059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:21.927286 17059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:21.927682 17081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:40:21.929124 17059 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:40:21.929170 17059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:40:21.929203 17059 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:40:21.929792 17059 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:40:21.929822 17059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:40:21.929893 17081 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Bootstrap starting.
I20260924 19:40:21.935304 17059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260924 19:40:21.935385 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260924 19:40:21.935671 17059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:21.936787 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.936311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:54347)
0924 19:40:21.936490 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.936598 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
W20260924 19:40:21.937033 17189 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37969 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37969: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
I20260924 19:40:21.937289 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.937197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:44253)
0924 19:40:21.937233 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.937265 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 19:40:21.937597 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.937524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:33571)
0924 19:40:21.937553 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.937577 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 19:40:21.937698 17189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:40:21.939489 17081 log.cc:824] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:21.941466 17081 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:21.941702 17081 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Bootstrap complete.
I20260924 19:40:21.942430 17081 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260924 19:40:21.943576 17081 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:21.943874 17081 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6037807dd14115bacd64760efb3e18, State: Initialized, Role: FOLLOWER
I20260924 19:40:21.944011 17081 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:21.944231 17081 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260924 19:40:21.944484 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.944394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:48089)
0924 19:40:21.944432 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.944462 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:40:21.944726 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:21.944661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:37459)
0924 19:40:21.944694 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:21.944713 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:40:21.945772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17059
I20260924 19:40:21.946018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41435
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37969
--webserver_port=37927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:22.025288 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:22.025493 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:22.025514 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:22.026592 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:22.026623 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:22.026648 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:22.028024 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37927
--tserver_master_addrs=127.0.98.62:37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:22.028203 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:22.028383 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:22.033051 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:22.033038 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:22.033209 17194 server_base.cc:1034] running on GCE node
W20260924 19:40:22.033025 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:22.033548 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:22.033761 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:22.034940 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790278822034920 us; error 26 us; skew 500 ppm
Sep 24 19:40:22 dist-test-slave-6kf3 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:22.037298 17194 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:40:22.037714 17194 webserver.cc:460] Webserver started at http://127.0.98.3:37927/ using document root <none> and password file <none>
I20260924 19:40:22.037904 17194 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:22.037952 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:22.039170 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:22.039654 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:22.039794 17194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:22.039866 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "38e0f00131e143cf897cd8ba49b3582f"
format_stamp: "Formatted at 2026-09-24 19:39:40 on dist-test-slave-6kf3"
server_key: "84da579408e5190afa1e6e32645e8566"
server_key_iv: "ff465d5952a69d25961980a4cc34dbe6"
server_key_version: "encryptionkey@0"
I20260924 19:40:22.040254 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:22.051800 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:22.053707 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:22.053844 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:22.054081 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:22.054447 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:22.054486 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:22.054515 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:22.054539 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:22.060226 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41435
I20260924 19:40:22.060273 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41435 every 8 connection(s)
I20260924 19:40:22.060606 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:40:22.061534 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:22.061167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:57735)
0924 19:40:22.061301 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:22.061380 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260924 19:40:22.061926 17323 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37969 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37969: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
I20260924 19:40:22.062215 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:22.062132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:36647)
0924 19:40:22.062172 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:22.062198 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:40:22.062439 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:22.062380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:44807)
0924 19:40:22.062408 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:22.062426 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:40:22.062526 17323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:40:22.070627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17194
I20260924 19:40:22.867311 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:22.867086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:38101)
0924 19:40:22.867212 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:22.867251 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260924 19:40:22.945221 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:22.945036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:55291)
0924 19:40:22.945139 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:22.945172 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260924 19:40:23.063015 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.062820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:55283)
0924 19:40:23.062933 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.062967 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260924 19:40:23.335780 17330 raft_consensus.cc:482] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:23.335914 17330 raft_consensus.cc:504] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:23.336257 17330 leader_election.cc:288] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435), 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
Sep 24 19:40:23 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 19:40:23 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:40:23.345810 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.336549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39699 (local address 127.0.98.1:45099)
0924 19:40:23.336834 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:23.336837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:23.336845 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:40:23.336912 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:23.336913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:23.336930 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:23.336997 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:23.337792 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.339220 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.339875 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.340304 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.340398 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:23.342866 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:23.342871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:23.342871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:23.342882 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:23.344538 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344816 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:23.344819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:23.344819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:23.344912 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:23.345074 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:23.345075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:23.345075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:23.345292 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:23.345356 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:23.345583 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:40:23.345692 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":74,"threads_started":1}
I20260924 19:40:23.345827 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.336408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45099 (local address 127.0.98.2:39699)
0924 19:40:23.336549 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.336559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:23.336704 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:23.337058 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:23.337063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:23.337447 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:23.337691 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:23.337697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.339312 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.339313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:23.339803 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:23.339805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.340382 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.340383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:23.340492 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:23.341038 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:23.341044 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:23.342780 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:23.344601 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:23.344603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:23.344752 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:23.344968 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:23.344969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:23.345026 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:23.345351 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:23.345352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:23.345439 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 19:40:23.345516 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:40:23.345700 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":11,"thread_start_us":53,"threads_started":1}
I20260924 19:40:23.345853 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.336546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51977 (local address 127.0.98.3:41435)
0924 19:40:23.336752 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:23.336763 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:23.336796 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:40:23.337352 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:23.337355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:23.337401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:23.337533 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:23.337793 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.338768 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.339837 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.340300 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.340393 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:23.342566 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:23.342571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:23.342571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:23.342584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:23.344340 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344719 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:23.344722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:23.344722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:23.344811 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:23.345032 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:23.345033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:23.345033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:23.345260 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:23.345328 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:23.345575 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:40:23.345743 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":59,"threads_started":1}
I20260924 19:40:23.346289 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.336407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41435 (local address 127.0.98.2:51977)
0924 19:40:23.336446 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.336473 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:23.336701 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:23.337517 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:23.337518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:23.337656 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:23.337705 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:23.337707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.338849 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.338850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:23.339760 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:23.339762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:23.340378 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:23.340379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:23.340490 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:23.341026 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:23.341033 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:23.342448 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:23.344429 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:23.344436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:23.344644 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:23.344866 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:23.344867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:23.344867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:23.344957 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:23.345318 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:23.345323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:23.345433 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:40:23.345516 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:40:23.345767 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":121}
I20260924 19:40:23.346581 17009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c" is_pre_election: true
I20260924 19:40:23.346716 17009 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 1.
I20260924 19:40:23.346951 17076 leader_election.cc:302] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6037807dd14115bacd64760efb3e18, 7494854bcfde48deaf02eb8f7bdc111c; no voters:
W20260924 19:40:23.347041 17075 leader_election.cc:341] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435): Not found: Tablet not found: 51dacaf7ad2043eb876c3532190532bf
I20260924 19:40:23.347069 17330 raft_consensus.cc:2781] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:40:23.347113 17330 raft_consensus.cc:482] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:23.347141 17330 raft_consensus.cc:3037] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:40:23.348235 17330 raft_consensus.cc:504] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:23.348358 17330 leader_election.cc:288] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 election: Requested vote from peers 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435), 7494854bcfde48deaf02eb8f7bdc111c (127.0.98.1:45099)
I20260924 19:40:23.348526 17009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dacaf7ad2043eb876c3532190532bf" candidate_uuid: "2e6037807dd14115bacd64760efb3e18" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7494854bcfde48deaf02eb8f7bdc111c"
W20260924 19:40:23.348585 17075 leader_election.cc:341] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435): Not found: Tablet not found: 51dacaf7ad2043eb876c3532190532bf
I20260924 19:40:23.348601 17009 raft_consensus.cc:3037] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:40:23.349677 17009 raft_consensus.cc:2454] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6037807dd14115bacd64760efb3e18 in term 2.
I20260924 19:40:23.349834 17076 leader_election.cc:302] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e6037807dd14115bacd64760efb3e18, 7494854bcfde48deaf02eb8f7bdc111c; no voters: 38e0f00131e143cf897cd8ba49b3582f
I20260924 19:40:23.349913 17330 raft_consensus.cc:2781] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:40:23.350049 17330 raft_consensus.cc:686] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [term 2 LEADER]: Becoming Leader. State: Replica: 2e6037807dd14115bacd64760efb3e18, State: Running, Role: LEADER
I20260924 19:40:23.350124 17330 consensus_queue.cc:237] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:23.350332 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.350247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:54963)
0924 19:40:23.350285 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.350307 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:40:23.806178 17009 raft_consensus.cc:1261] T 51dacaf7ad2043eb876c3532190532bf P 7494854bcfde48deaf02eb8f7bdc111c [term 2 FOLLOWER]: Refusing update from remote peer 2e6037807dd14115bacd64760efb3e18: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 19:40:23.806540 17330 consensus_queue.cc:1035] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 19:40:23.807461 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.807264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:36507)
0924 19:40:23.807349 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.807417 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":45,"threads_started":1}
I20260924 19:40:23.808280 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.808139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:59469)
0924 19:40:23.808216 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.808246 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:40:23.808333 17075 consensus_peers.cc:487] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18 -> Peer 38e0f00131e143cf897cd8ba49b3582f (127.0.98.3:41435): Couldn't send request to peer 38e0f00131e143cf897cd8ba49b3582f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51dacaf7ad2043eb876c3532190532bf. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:40:23.808504 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:23.808394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:49543)
0924 19:40:23.808456 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:23.808478 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 19:40:24.063580 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.063376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:37031)
0924 19:40:24.063512 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.063543 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260924 19:40:24.221102 17343 ts_tablet_manager.cc:927] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Initiating tablet copy from peer 2e6037807dd14115bacd64760efb3e18 (127.0.98.2:37875)
I20260924 19:40:24.221342 17343 tablet_copy_client.cc:316] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:37875
Sep 24 19:40:24 dist-test-slave-6kf3 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 19:40:24.228612 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.221694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42615 (local address 127.0.98.2:37875)
0924 19:40:24.221800 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:24.221804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:24.221813 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:24.221854 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:24.221856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:24.221868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:24.221965 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:24.222377 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:24.223072 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:24.223757 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:24.224080 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:24.224128 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:24.226213 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:24.226221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:24.226221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:24.226227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:24.227619 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:24.227986 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:24.227988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:24.227989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:24.228041 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:24.228189 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:24.228189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:24.228189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:24.228315 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:24.228369 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:24.228487 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 19:40:24.228544 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":43,"threads_started":1}
I20260924 19:40:24.228632 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.221613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.98.3:42615)
0924 19:40:24.221665 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.221680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:24.221734 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:24.221971 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:24.221974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:24.222246 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:24.222312 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:24.222314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:24.223131 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:24.223131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:24.223703 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:24.223705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:24.224149 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:24.224151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:24.224235 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:24.224755 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:24.224760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:24.226142 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:24.227692 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:24.227698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:24.227699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:24.227878 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:24.228091 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:24.228092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:24.228092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:24.228146 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:24.228356 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:24.228361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:24.228404 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:40:24.228458 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:40:24.228556 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:40:24.229034 17163 tablet_copy_service.cc:140] P 2e6037807dd14115bacd64760efb3e18: Received BeginTabletCopySession request for tablet 51dacaf7ad2043eb876c3532190532bf from peer 38e0f00131e143cf897cd8ba49b3582f ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42615)
I20260924 19:40:24.229099 17163 tablet_copy_service.cc:161] P 2e6037807dd14115bacd64760efb3e18: Beginning new tablet copy session on tablet 51dacaf7ad2043eb876c3532190532bf from peer 38e0f00131e143cf897cd8ba49b3582f at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42615: session id = 38e0f00131e143cf897cd8ba49b3582f-51dacaf7ad2043eb876c3532190532bf
I20260924 19:40:24.229516 17163 tablet_copy_source_session.cc:205] T 51dacaf7ad2043eb876c3532190532bf P 2e6037807dd14115bacd64760efb3e18: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:40:24.230435 17343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dacaf7ad2043eb876c3532190532bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:24.233078 17343 tablet_copy_client.cc:791] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: tablet copy: Starting download of 0 data blocks...
I20260924 19:40:24.233215 17343 tablet_copy_client.cc:655] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: tablet copy: Starting download of 1 WAL segments...
I20260924 19:40:24.233919 17343 tablet_copy_client.cc:531] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:40:24.234562 17343 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Bootstrap starting.
I20260924 19:40:24.235821 17343 log.cc:824] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:24.238219 17343 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:40:24.238431 17343 tablet_bootstrap.cc:492] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Bootstrap complete.
I20260924 19:40:24.238551 17343 ts_tablet_manager.cc:1397] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 19:40:24.239480 17343 raft_consensus.cc:348] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:24.239629 17343 raft_consensus.cc:729] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0f00131e143cf897cd8ba49b3582f, State: Initialized, Role: FOLLOWER
I20260924 19:40:24.239753 17343 consensus_queue.cc:260] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6037807dd14115bacd64760efb3e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "38e0f00131e143cf897cd8ba49b3582f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41435 } } peers { permanent_uuid: "7494854bcfde48deaf02eb8f7bdc111c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45099 } }
I20260924 19:40:24.240026 17343 ts_tablet_manager.cc:1428] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f: Time spent starting tablet: real 0.001s user 0.001s sys 0.003s
I20260924 19:40:24.240268 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.240181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:42269)
0924 19:40:24.240218 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.240239 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:40:24.240320 17163 tablet_copy_service.cc:342] P 2e6037807dd14115bacd64760efb3e18: Request end of tablet copy session 38e0f00131e143cf897cd8ba49b3582f-51dacaf7ad2043eb876c3532190532bf received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42615
I20260924 19:40:24.240384 17163 tablet_copy_service.cc:433] P 2e6037807dd14115bacd64760efb3e18: ending tablet copy session 38e0f00131e143cf897cd8ba49b3582f-51dacaf7ad2043eb876c3532190532bf on tablet 51dacaf7ad2043eb876c3532190532bf with peer 38e0f00131e143cf897cd8ba49b3582f
I20260924 19:40:24.240567 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.240470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:37773)
0924 19:40:24.240508 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.240542 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 19:40:24.717450 17277 raft_consensus.cc:1203] T 51dacaf7ad2043eb876c3532190532bf P 38e0f00131e143cf897cd8ba49b3582f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 19:40:24.717499 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.717327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:38395)
0924 19:40:24.717422 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.717457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260924 19:40:24.717942 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.717786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:59365)
0924 19:40:24.717835 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.717912 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:40:24.809084 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:24.808899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:39655)
0924 19:40:24.809003 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:24.809039 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260924 19:40:25.718139 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:25.717902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:44139)
0924 19:40:25.718040 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:25.718084 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260924 19:40:25.718502 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:25.718341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:49337)
0924 19:40:25.718436 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:25.718464 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
I20260924 19:40:25.809677 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:25.809492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:44741)
0924 19:40:25.809600 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:25.809632 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260924 19:40:26.718881 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:26.718638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:42127)
0924 19:40:26.718793 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:26.718835 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260924 19:40:26.719087 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:26.718900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:37417)
0924 19:40:26.719003 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:26.719045 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":43,"threads_started":1}
I20260924 19:40:26.810326 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:26.810130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:42865)
0924 19:40:26.810241 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:26.810279 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260924 19:40:27.719548 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:27.719325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:60589)
0924 19:40:27.719453 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:27.719493 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260924 19:40:27.719744 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:27.719541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:35371)
0924 19:40:27.719660 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:27.719699 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 19:40:27.720019 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:27.719867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:46067)
0924 19:40:27.719914 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:27.719994 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:40:27.811007 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:27.810783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:53801)
0924 19:40:27.810917 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:27.810958 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260924 19:40:28.720337 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:28.720153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:44595)
0924 19:40:28.720265 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:28.720294 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 19:40:28.720551 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:28.720346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:54617)
0924 19:40:28.720476 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:28.720509 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":43,"threads_started":1}
I20260924 19:40:28.811650 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:28.811456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:56999)
0924 19:40:28.811569 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:28.811604 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260924 19:40:29.721084 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:29.720845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:49531)
0924 19:40:29.720982 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:29.721025 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 19:40:29.721163 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:29.720945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:44547)
0924 19:40:29.721084 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:29.721122 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260924 19:40:29.812341 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:29.812121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:60615)
0924 19:40:29.812259 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:29.812296 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260924 19:40:30.721761 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:30.721553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:37483)
0924 19:40:30.721687 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:30.721721 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260924 19:40:30.721756 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:30.721506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:39523)
0924 19:40:30.721664 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:30.721709 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260924 19:40:30.812973 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:30.812749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:55873)
0924 19:40:30.812894 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:30.812928 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 19:40:31.722451 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:31.722225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:56891)
0924 19:40:31.722371 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:31.722406 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260924 19:40:31.722512 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:31.722259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:59875)
0924 19:40:31.722436 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:31.722472 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260924 19:40:31.813761 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:31.813529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:59167)
0924 19:40:31.813663 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:31.813703 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260924 19:40:32.723178 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:32.722955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:56721)
0924 19:40:32.723108 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:32.723138 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260924 19:40:32.723173 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:32.722955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:38647)
0924 19:40:32.723086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:32.723128 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260924 19:40:32.814404 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:32.814196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:54731)
0924 19:40:32.814320 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:32.814358 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 19:40:33.723803 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:33.723604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:37019)
0924 19:40:33.723729 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:33.723757 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260924 19:40:33.723803 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:33.723605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:58035)
0924 19:40:33.723729 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:33.723761 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 19:40:33.815025 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:33.814786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:36027)
0924 19:40:33.814895 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:33.814964 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260924 19:40:34.724534 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:34.724306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:58321)
0924 19:40:34.724454 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:34.724489 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260924 19:40:34.724551 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:34.724306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:56897)
0924 19:40:34.724464 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:34.724510 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260924 19:40:34.815727 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:34.815538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:33145)
0924 19:40:34.815660 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:34.815690 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260924 19:40:35.725291 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:35.725055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:51897)
0924 19:40:35.725192 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:35.725236 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260924 19:40:35.725291 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:35.725055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:33745)
0924 19:40:35.725210 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:35.725241 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260924 19:40:35.816432 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:35.816198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:33279)
0924 19:40:35.816333 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:35.816381 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260924 19:40:36.725962 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:36.725762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.3:41627)
0924 19:40:36.725882 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:36.725915 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260924 19:40:36.725973 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:36.725773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.2:45399)
0924 19:40:36.725894 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:36.725925 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260924 19:40:36.817121 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:36.816899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37969 (local address 127.0.98.1:45875)
0924 19:40:36.817043 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:36.817080 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260924 19:40:37.079766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 16890
I20260924 19:40:37.083889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17059
I20260924 19:40:37.087780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17194
2026-09-24T19:40:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58162 ms)
[----------] 30 tests from SecurityITest (252447 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:37 dist-test-slave-6kf3 krb5kdc[17392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:37 dist-test-slave-6kf3 krb5kdc[17392](info): set up 2 sockets
Sep 24 19:40:37 dist-test-slave-6kf3 krb5kdc[17392](info): commencing operation
krb5kdc: starting...
W20260924 19:40:38.039477 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:40:38.151249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:38.230643 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:38.230857 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:38.230880 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:38.232013 17405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:38.232049 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:38.232062 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:38.232074 17405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:38.232084 17405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:38.233492 17405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:38.233703 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:38.233876 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:38.238894 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.238950 17405 server_base.cc:1034] running on GCE node
W20260924 19:40:38.238926 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:38.239049 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.239392 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:38.239715 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:38.240877 17405 hybrid_clock.cc:648] HybridClock initialized: now 1790278838240856 us; error 32 us; skew 500 ppm
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:38.243381 17405 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:40:38.243880 17405 webserver.cc:460] Webserver started at http://127.0.98.62:43241/ using document root <none> and password file <none>
I20260924 19:40:38.244160 17405 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:38.244215 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:38.244345 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:38.245510 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "3ae37b13234f40a2ba3dc69fe6fc1892"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "d6943acf587e04cac0ff56ff8c14b4b7"
server_key_iv: "6052935f8fe3863faeb2883fbf278c7e"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.245867 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ae37b13234f40a2ba3dc69fe6fc1892"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "d6943acf587e04cac0ff56ff8c14b4b7"
server_key_iv: "6052935f8fe3863faeb2883fbf278c7e"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.247213 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:38.247882 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.248049 17405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:38.248129 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "3ae37b13234f40a2ba3dc69fe6fc1892"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "d6943acf587e04cac0ff56ff8c14b4b7"
server_key_iv: "6052935f8fe3863faeb2883fbf278c7e"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.248200 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:38.267447 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:38.269330 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:38.269471 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:38.273379 17405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45091
I20260924 19:40:38.273419 17472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45091 every 8 connection(s)
I20260924 19:40:38.273754 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:38.274590 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:38.275720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17405
I20260924 19:40:38.275818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:38.276046 392 external_mini_cluster.cc:1428] Setting key fcbe10e572542ee0ead57cd5a63e9e9d
I20260924 19:40:38.277226 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: Bootstrap starting.
I20260924 19:40:38.277899 17473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:38.278224 17473 log.cc:824] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:38.278920 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: No bootstrap required, opened a new log
I20260924 19:40:38.281180 17473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260924 19:40:38.281322 17473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:38.281353 17473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae37b13234f40a2ba3dc69fe6fc1892, State: Initialized, Role: FOLLOWER
I20260924 19:40:38.281486 17473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260924 19:40:38.281558 17473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:38.281587 17473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:38.281628 17473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:38.282157 17473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260924 19:40:38.282253 17473 leader_election.cc:302] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ae37b13234f40a2ba3dc69fe6fc1892; no voters:
I20260924 19:40:38.282407 17473 leader_election.cc:288] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:38.282447 17478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:38.282586 17478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae37b13234f40a2ba3dc69fe6fc1892, State: Running, Role: LEADER
I20260924 19:40:38.282702 17473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:38.282716 17478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:38.283109 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260924 19:40:38.283202 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:38.283285 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ae37b13234f40a2ba3dc69fe6fc1892. Latest consensus state: current_term: 1 leader_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae37b13234f40a2ba3dc69fe6fc1892" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260924 19:40:38.283334 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892 [sys.catalog]: This master's current role is: LEADER
W20260924 19:40:38.284682 17494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:40:38.284734 17494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:40:38.284843 17493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:38.285059 17493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:38.287209 17493 catalog_manager.cc:1347] Generated new cluster ID: 538522e83ecc44a1a31781d8164668ff
I20260924 19:40:38.287261 17493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:38.287288 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.276585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.62:45091)
0924 19:40:38.276738 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.276749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.276782 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:40:38.277242 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.277249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.277309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.277443 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:38.278178 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.279248 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.280332 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.280771 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.280872 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.283620 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:38.283624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:38.283624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:38.283636 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:38.285597 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.285967 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.285971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.285971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.286093 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.286336 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.286337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.286338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.286642 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:38.286804 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.286926 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.287072 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 19:40:38.295744 17493 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:38.296312 17493 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:38.303781 17493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: Generated new TSK 0
I20260924 19:40:38.304006 17493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:40:38.330802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:38.409901 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:38.410125 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:38.410149 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:38.411279 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:38.411320 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:38.411351 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:38.412850 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:38.413069 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:38.413254 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:38.418063 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.418176 17499 server_base.cc:1034] running on GCE node
W20260924 19:40:38.418102 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:38.417982 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.418457 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:38.418704 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:38.419842 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790278838419832 us; error 27 us; skew 500 ppm
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:38.422307 17499 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:40:38.422704 17499 webserver.cc:460] Webserver started at http://127.0.98.1:33321/ using document root <none> and password file <none>
I20260924 19:40:38.422904 17499 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:38.422946 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:38.423079 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:38.424145 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "943e6ac46c7141838e19348cad5d01c5"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "c790f0090160db7577229f181338d2e5"
server_key_iv: "3677c8f0289db17be21d492ee52d44ab"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.424455 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "943e6ac46c7141838e19348cad5d01c5"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "c790f0090160db7577229f181338d2e5"
server_key_iv: "3677c8f0289db17be21d492ee52d44ab"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.425686 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:38.426496 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.426631 17499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:38.426707 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "943e6ac46c7141838e19348cad5d01c5"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "c790f0090160db7577229f181338d2e5"
server_key_iv: "3677c8f0289db17be21d492ee52d44ab"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.426769 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:38.436038 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:38.438107 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:38.438220 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:38.438438 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:38.438735 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:38.438769 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.438791 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:38.438802 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.444103 17499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44585
I20260924 19:40:38.444144 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44585 every 8 connection(s)
I20260924 19:40:38.444497 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:38.445124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17499
I20260924 19:40:38.445217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:38.445411 392 external_mini_cluster.cc:1428] Setting key edbada232b4af15f5d08b5323912f8cf
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:40:38.454645 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.445410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48375 (local address 127.0.98.62:45091)
0924 19:40:38.445477 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.445479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.445715 (+ 236us) server_negotiation.cc:407] Connection header received
0924 19:40:38.446766 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.446768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.446785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.446850 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:38.447531 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.448089 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.449457 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.449671 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.449723 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.452123 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:38.452124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:38.452125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:38.452133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:38.453425 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.453852 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.453853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.453854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.453903 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.454122 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.454123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.454123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.454223 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:38.454268 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.454507 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.454576 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:40:38.454784 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.445406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.1:48375)
0924 19:40:38.445581 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.445630 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.446659 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.446904 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.446907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.447165 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:38.447439 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.447445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.448170 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.448172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.449370 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.449374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.449752 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.449753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.449902 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.450406 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:38.450416 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:38.452042 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:38.453516 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.453524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.453526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.453790 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.453955 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.453956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.453956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.454076 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.454272 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:38.454277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:38.454407 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.454483 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.454601 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260924 19:40:38.455289 17628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260924 19:40:38.455386 17628 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 19:40:38.455578 17628 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260924 19:40:38.456100 17436 ts_manager.cc:194] Registered new tserver with Master: 943e6ac46c7141838e19348cad5d01c5 (127.0.98.1:44585)
I20260924 19:40:38.456756 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:40:38.470320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:38.548866 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:38.549055 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:38.549077 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:38.550156 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:38.550189 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:38.550212 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:38.551657 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:38.551837 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:38.552027 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:38.556663 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:38.556639 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.556730 17633 server_base.cc:1034] running on GCE node
W20260924 19:40:38.556636 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.557116 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:38.557334 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:38.558494 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790278838558475 us; error 32 us; skew 500 ppm
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:38.560894 17633 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:40:38.561308 17633 webserver.cc:460] Webserver started at http://127.0.98.2:42961/ using document root <none> and password file <none>
I20260924 19:40:38.561518 17633 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:38.561564 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:38.561685 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:38.562675 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6f32728d20e43edacb85ac48dbbc374"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "45beec0553a4466f6f9a87e2b98ca922"
server_key_iv: "93863195f743a33349365ad257f940de"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.562992 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6f32728d20e43edacb85ac48dbbc374"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "45beec0553a4466f6f9a87e2b98ca922"
server_key_iv: "93863195f743a33349365ad257f940de"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.564206 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:38.564845 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.564992 17633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:38.565063 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "b6f32728d20e43edacb85ac48dbbc374"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "45beec0553a4466f6f9a87e2b98ca922"
server_key_iv: "93863195f743a33349365ad257f940de"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.565127 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:38.596015 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:38.598217 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:38.598383 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:38.598590 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:38.598881 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:38.598912 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.598932 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:38.598943 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.604233 17633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44951
I20260924 19:40:38.604288 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44951 every 8 connection(s)
I20260924 19:40:38.604580 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:38.604879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17633
I20260924 19:40:38.604985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:38.605180 392 external_mini_cluster.cc:1428] Setting key 6f94c62f798e6c4545b0adc893a68308
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:40:38.615044 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.605318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57297 (local address 127.0.98.62:45091)
0924 19:40:38.605429 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.605431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.605631 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:40:38.606615 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.606617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.606628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.606684 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:38.607497 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.608033 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.609472 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.609683 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.609732 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.612307 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:38.612308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:38.612309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:38.612316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:38.613641 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.614038 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.614040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.614041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.614107 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.614354 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.614355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.614355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.614480 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:38.614567 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.614857 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.614952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:40:38.615208 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.605253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.2:57297)
0924 19:40:38.605501 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.605547 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.606494 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.606759 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.606763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.607125 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:38.607399 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.607405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.608119 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.608120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.609382 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.609385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.609761 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.609762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.609905 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.610404 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:38.610414 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:38.612219 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:38.613719 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.613726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.613727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.613938 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.614173 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.614173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.614174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.614276 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.614581 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:38.614588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:38.614727 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.614830 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.614992 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 19:40:38.615644 17762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260924 19:40:38.615749 17762 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:38.615957 17762 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260924 19:40:38.616360 17436 ts_manager.cc:194] Registered new tserver with Master: b6f32728d20e43edacb85ac48dbbc374 (127.0.98.2:44951)
I20260924 19:40:38.616719 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:40:38.628643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:38.708541 17767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:38.708750 17767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:38.708781 17767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:38.709904 17767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:38.709951 17767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:38.709987 17767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:38.711447 17767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:38.711676 17767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:38.711882 17767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:38.716616 17773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:38.716616 17772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.716699 17767 server_base.cc:1034] running on GCE node
W20260924 19:40:38.716616 17775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:38.717037 17767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:38.717308 17767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:38.718454 17767 hybrid_clock.cc:648] HybridClock initialized: now 1790278838718441 us; error 28 us; skew 500 ppm
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:38.721035 17767 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:40:38.721481 17767 webserver.cc:460] Webserver started at http://127.0.98.3:33169/ using document root <none> and password file <none>
I20260924 19:40:38.721683 17767 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:38.721732 17767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:38.721856 17767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:38.722909 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "451d276cff484641aea55756f9d61a35"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "26cb8c44bd44d9b9b78bc9383706812f"
server_key_iv: "6dba57b2f5c75b2ef1d769083a6f1347"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.723253 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "451d276cff484641aea55756f9d61a35"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "26cb8c44bd44d9b9b78bc9383706812f"
server_key_iv: "6dba57b2f5c75b2ef1d769083a6f1347"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.724571 17767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:38.725379 17782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.725519 17767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:38.725602 17767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "451d276cff484641aea55756f9d61a35"
format_stamp: "Formatted at 2026-09-24 19:40:38 on dist-test-slave-6kf3"
server_key: "26cb8c44bd44d9b9b78bc9383706812f"
server_key_iv: "6dba57b2f5c75b2ef1d769083a6f1347"
server_key_version: "encryptionkey@0"
I20260924 19:40:38.725664 17767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:38.734792 17767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:38.736644 17767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:38.736778 17767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:38.736996 17767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:38.737278 17767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:38.737310 17767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.737339 17767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:38.737363 17767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:38.742950 17767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40617
I20260924 19:40:38.743009 17895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40617 every 8 connection(s)
I20260924 19:40:38.743294 17767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:40:38 dist-test-slave-6kf3 krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:38.752231 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.743986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56567 (local address 127.0.98.62:45091)
0924 19:40:38.744054 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.744056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.744201 (+ 145us) server_negotiation.cc:407] Connection header received
0924 19:40:38.745164 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.745167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.745179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.745229 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:38.745831 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.746347 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.747404 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.747608 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.747663 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.749717 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:38.749719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:38.749719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:38.749727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:38.750868 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.751300 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.751301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.751302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.751352 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.751583 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.751584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.751584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.751703 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:38.751758 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.752059 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.752139 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:40:38.752404 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.743911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.3:56567)
0924 19:40:38.744086 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.744123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.745056 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.745297 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.745300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.745551 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:38.745741 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.745746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.746441 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.746441 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.747317 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.747319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.747688 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.747689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.747805 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.748328 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:38.748336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:38.749626 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:38.750990 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.750996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.750997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.751226 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.751428 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:38.751428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:38.751429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:38.751525 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:38.751763 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:38.751770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:38.751902 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.752037 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.752191 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260924 19:40:38.752938 17896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260924 19:40:38.753041 17896 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:38.753060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17767
I20260924 19:40:38.753167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:38.753262 17896 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260924 19:40:38.753371 392 external_mini_cluster.cc:1428] Setting key 0ce1a66e976ef3939da1e3121d2cab05
I20260924 19:40:38.753682 17436 ts_manager.cc:194] Registered new tserver with Master: 451d276cff484641aea55756f9d61a35 (127.0.98.3:40617)
I20260924 19:40:38.754020 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56567
I20260924 19:40:38.754722 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:38.761790 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.755982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.98.62:45091)
0924 19:40:38.756015 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.756017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.756152 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:40:38.756189 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.756190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.756198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.756247 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:38.756740 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.757250 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.758002 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.758183 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.758243 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.759338 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:38.759339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:38.759339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:38.759345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:38.760663 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.761041 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.761042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.761043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.761097 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:38.761300 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:38.761301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:38.761302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:38.761452 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:38.761532 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.761670 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.761727 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:40:38.763348 17436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:38.765026 17436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:38.771203 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.767273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.0.98.3:40617)
0924 19:40:38.767408 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.767411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.767600 (+ 189us) server_negotiation.cc:407] Connection header received
0924 19:40:38.767660 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.767662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.767676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.767746 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:38.767911 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.769109 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770313 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770831 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770892 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.770958 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.771088 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.771127 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260924 19:40:38.771229 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.767175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40617 (local address 127.0.0.1:46922)
0924 19:40:38.767496 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.767504 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.767546 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.767753 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.767754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.767756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:38.767828 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.767830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.769205 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.769205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.770243 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.770245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770913 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.770989 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.771002 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.771067 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.771120 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":25,"thread_start_us":66,"threads_started":1}
I20260924 19:40:38.771272 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.767114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.0.98.2:44951)
0924 19:40:38.767324 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.767329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.767524 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:40:38.767570 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.767572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.767605 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.767672 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:38.767993 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.769065 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770107 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770839 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770886 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.770943 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.771162 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.771199 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260924 19:40:38.771317 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.766976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44585 (local address 127.0.0.1:43176)
0924 19:40:38.767292 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.767336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.767398 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.767707 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.767710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.767712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:38.767774 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.767776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.768828 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.768829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.770053 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.770056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770983 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.771072 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.771082 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.771122 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.771160 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260924 19:40:38.771379 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.767037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.0.98.1:44585)
0924 19:40:38.767230 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.767235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.767398 (+ 163us) server_negotiation.cc:407] Connection header received
0924 19:40:38.767486 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.767488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.767505 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.767610 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:38.767847 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.768742 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770123 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770841 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770905 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.770950 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.771268 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.771310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260924 19:40:38.771623 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.766976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44951 (local address 127.0.0.1:34778)
0924 19:40:38.767434 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.767443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.767511 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.767753 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.767754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.767758 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:38.767872 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.767873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.769181 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.769182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.770023 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.770026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.770995 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.770995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.771072 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.771078 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.771343 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.771380 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":89,"spinlock_wait_cycles":384,"thread_start_us":64,"threads_started":1}
I20260924 19:40:38.772070 17562 tablet_service.cc:1467] Processing CreateTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 (DEFAULT_TABLE table=test-table [id=24b21b43a9964cdb839b250c87e25e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:38.772426 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c051bc3c1843769ba56ca32d2fc562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:38.772375 17696 tablet_service.cc:1467] Processing CreateTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 (DEFAULT_TABLE table=test-table [id=24b21b43a9964cdb839b250c87e25e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:38.772495 17830 tablet_service.cc:1467] Processing CreateTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 (DEFAULT_TABLE table=test-table [id=24b21b43a9964cdb839b250c87e25e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:38.772691 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c051bc3c1843769ba56ca32d2fc562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:38.772765 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c051bc3c1843769ba56ca32d2fc562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:38.775035 17910 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Bootstrap starting.
I20260924 19:40:38.775602 17912 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Bootstrap starting.
I20260924 19:40:38.775637 17910 tablet_bootstrap.cc:654] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:38.776223 17912 tablet_bootstrap.cc:654] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:38.776219 17910 log.cc:824] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:38.776697 17912 log.cc:824] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:38.776976 17910 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: No bootstrap required, opened a new log
I20260924 19:40:38.777036 17911 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: Bootstrap starting.
I20260924 19:40:38.777101 17910 ts_tablet_manager.cc:1397] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:40:38.777328 17912 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: No bootstrap required, opened a new log
I20260924 19:40:38.777388 17912 ts_tablet_manager.cc:1397] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:40:38.777630 17911 tablet_bootstrap.cc:654] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:38.778132 17911 log.cc:824] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:38.778920 17911 tablet_bootstrap.cc:492] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: No bootstrap required, opened a new log
I20260924 19:40:38.779000 17911 ts_tablet_manager.cc:1397] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:38.779043 17910 raft_consensus.cc:348] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.779073 17912 raft_consensus.cc:348] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.779217 17912 raft_consensus.cc:374] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:38.779249 17910 raft_consensus.cc:374] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:38.779275 17912 raft_consensus.cc:729] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f32728d20e43edacb85ac48dbbc374, State: Initialized, Role: FOLLOWER
I20260924 19:40:38.779309 17910 raft_consensus.cc:729] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451d276cff484641aea55756f9d61a35, State: Initialized, Role: FOLLOWER
I20260924 19:40:38.779402 17912 consensus_queue.cc:260] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.779455 17910 consensus_queue.cc:260] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.779666 17762 heartbeater.cc:503] Master 127.0.98.62:45091 was elected leader, sending a full tablet report...
I20260924 19:40:38.779675 17912 ts_tablet_manager.cc:1428] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:40:38.779724 17910 ts_tablet_manager.cc:1428] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:40:38.779832 17896 heartbeater.cc:503] Master 127.0.98.62:45091 was elected leader, sending a full tablet report...
I20260924 19:40:38.781067 17911 raft_consensus.cc:348] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.781207 17911 raft_consensus.cc:374] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:38.781230 17911 raft_consensus.cc:729] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943e6ac46c7141838e19348cad5d01c5, State: Initialized, Role: FOLLOWER
I20260924 19:40:38.781320 17911 consensus_queue.cc:260] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.781558 17911 ts_tablet_manager.cc:1428] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:40:38.781601 17628 heartbeater.cc:503] Master 127.0.98.62:45091 was elected leader, sending a full tablet report...
W20260924 19:40:38.855126 17763 tablet.cc:2369] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:38.904958 17918 raft_consensus.cc:482] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:38.905087 17918 raft_consensus.cc:504] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.905382 17918 leader_election.cc:288] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6f32728d20e43edacb85ac48dbbc374 (127.0.98.2:44951), 451d276cff484641aea55756f9d61a35 (127.0.98.3:40617)
I20260924 19:40:38.908345 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.905683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40617 (local address 127.0.98.1:43085)
0924 19:40:38.905836 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.905846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.905902 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.906063 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.906064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.906066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:38.906135 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.906136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.906731 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.906732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.907530 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.907532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.908121 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.908122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.908188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.908191 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.908230 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.908260 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260924 19:40:38.908367 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.905769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43085 (local address 127.0.98.3:40617)
0924 19:40:38.905837 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.905840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.905894 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:40:38.905976 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.905977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.905986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.906059 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:38.906199 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.906645 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.907631 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.908045 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.908113 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.908124 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.908257 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.908297 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:38.908401 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.905706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58635 (local address 127.0.98.2:44951)
0924 19:40:38.905759 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.905761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.905773 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:40:38.905859 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.905861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.905873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.905927 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:38.906104 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.906614 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.907641 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.908053 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.908112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.908123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.908302 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.908343 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:40:38.908541 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.905588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44951 (local address 127.0.98.1:58635)
0924 19:40:38.905677 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.905694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:38.905767 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:38.905952 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:38.905953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:38.905955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:38.906034 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.906036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.906701 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.906702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.907547 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:38.907549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.908138 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.908139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:38.908228 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.908232 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:38.908292 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:40:38.908331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":33}
I20260924 19:40:38.908882 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c051bc3c1843769ba56ca32d2fc562" candidate_uuid: "943e6ac46c7141838e19348cad5d01c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f32728d20e43edacb85ac48dbbc374" is_pre_election: true
I20260924 19:40:38.908897 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c051bc3c1843769ba56ca32d2fc562" candidate_uuid: "943e6ac46c7141838e19348cad5d01c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451d276cff484641aea55756f9d61a35" is_pre_election: true
I20260924 19:40:38.909045 17850 raft_consensus.cc:2454] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943e6ac46c7141838e19348cad5d01c5 in term 0.
I20260924 19:40:38.909046 17716 raft_consensus.cc:2454] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943e6ac46c7141838e19348cad5d01c5 in term 0.
I20260924 19:40:38.909265 17516 leader_election.cc:302] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 943e6ac46c7141838e19348cad5d01c5, b6f32728d20e43edacb85ac48dbbc374; no voters:
I20260924 19:40:38.909384 17918 raft_consensus.cc:2781] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:38.909440 17918 raft_consensus.cc:482] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:38.909462 17918 raft_consensus.cc:3037] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:38.910108 17918 raft_consensus.cc:504] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.910212 17918 leader_election.cc:288] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [CANDIDATE]: Term 1 election: Requested vote from peers b6f32728d20e43edacb85ac48dbbc374 (127.0.98.2:44951), 451d276cff484641aea55756f9d61a35 (127.0.98.3:40617)
I20260924 19:40:38.910387 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c051bc3c1843769ba56ca32d2fc562" candidate_uuid: "943e6ac46c7141838e19348cad5d01c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f32728d20e43edacb85ac48dbbc374"
I20260924 19:40:38.910436 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c051bc3c1843769ba56ca32d2fc562" candidate_uuid: "943e6ac46c7141838e19348cad5d01c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451d276cff484641aea55756f9d61a35"
I20260924 19:40:38.910470 17716 raft_consensus.cc:3037] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:38.910512 17850 raft_consensus.cc:3037] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:38.911123 17716 raft_consensus.cc:2454] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943e6ac46c7141838e19348cad5d01c5 in term 1.
I20260924 19:40:38.911248 17850 raft_consensus.cc:2454] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943e6ac46c7141838e19348cad5d01c5 in term 1.
I20260924 19:40:38.911278 17516 leader_election.cc:302] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 943e6ac46c7141838e19348cad5d01c5, b6f32728d20e43edacb85ac48dbbc374; no voters:
I20260924 19:40:38.911356 17918 raft_consensus.cc:2781] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:38.911514 17918 raft_consensus.cc:686] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 1 LEADER]: Becoming Leader. State: Replica: 943e6ac46c7141838e19348cad5d01c5, State: Running, Role: LEADER
I20260924 19:40:38.911633 17918 consensus_queue.cc:237] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } }
I20260924 19:40:38.912544 17437 catalog_manager.cc:5495] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 943e6ac46c7141838e19348cad5d01c5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "943e6ac46c7141838e19348cad5d01c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "943e6ac46c7141838e19348cad5d01c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 } health_report { overall_health: UNKNOWN } } }
I20260924 19:40:38.918164 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.915895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.0.98.1:44585)
0924 19:40:38.915997 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:38.915999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:38.916007 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:40:38.916034 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:38.916035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:38.916052 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:38.916106 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:38.916201 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.916647 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.917446 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:38.917644 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:38.917698 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:38.917872 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:38.917972 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:38.918027 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:38.918064 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 19:40:38.918079 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:40:38.919898 17850 raft_consensus.cc:1261] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 1 FOLLOWER]: Refusing update from remote peer 943e6ac46c7141838e19348cad5d01c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:38.919900 17716 raft_consensus.cc:1261] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 1 FOLLOWER]: Refusing update from remote peer 943e6ac46c7141838e19348cad5d01c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:38.920281 17920 consensus_queue.cc:1035] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6f32728d20e43edacb85ac48dbbc374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:38.920411 17918 consensus_queue.cc:1035] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451d276cff484641aea55756f9d61a35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:38.922719 17925 mvcc.cc:205] Tried to move back new op lower bound from 7332982124213219328 to 7332982124182622208. Current Snapshot: MvccSnapshot[applied={T|T < 7332982124213219328}]
I20260924 19:40:38.925081 17437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:38.925163 17437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:38.925876 17437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ae37b13234f40a2ba3dc69fe6fc1892: Sending DeleteTablet for 3 replicas of tablet f6c051bc3c1843769ba56ca32d2fc562
I20260924 19:40:38.926185 17562 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43176
I20260924 19:40:38.926232 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34778
I20260924 19:40:38.926492 17933 tablet_replica.cc:330] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5: stopping tablet replica
I20260924 19:40:38.926530 17830 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c051bc3c1843769ba56ca32d2fc562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46922
I20260924 19:40:38.926597 17933 raft_consensus.cc:2229] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:38.926698 17933 raft_consensus.cc:2258] T f6c051bc3c1843769ba56ca32d2fc562 P 943e6ac46c7141838e19348cad5d01c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:38.926731 17935 tablet_replica.cc:330] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: stopping tablet replica
I20260924 19:40:38.926823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17499
I20260924 19:40:38.927366 17934 tablet_replica.cc:330] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: stopping tablet replica
I20260924 19:40:38.927454 17934 raft_consensus.cc:2229] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:38.927543 17934 raft_consensus.cc:2258] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:38.926857 17935 raft_consensus.cc:2229] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:38.927991 17935 raft_consensus.cc:2258] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:38.927994 17934 ts_tablet_manager.cc:1905] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:38.928426 17935 ts_tablet_manager.cc:1905] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:38.929636 17934 ts_tablet_manager.cc:1918] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:38.929716 17934 log.cc:1196] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/f6c051bc3c1843769ba56ca32d2fc562
I20260924 19:40:38.929884 17935 ts_tablet_manager.cc:1918] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:38.929951 17935 log.cc:1196] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/f6c051bc3c1843769ba56ca32d2fc562
I20260924 19:40:38.929968 17934 ts_tablet_manager.cc:1939] T f6c051bc3c1843769ba56ca32d2fc562 P b6f32728d20e43edacb85ac48dbbc374: Deleting consensus metadata
I20260924 19:40:38.930209 17935 ts_tablet_manager.cc:1939] T f6c051bc3c1843769ba56ca32d2fc562 P 451d276cff484641aea55756f9d61a35: Deleting consensus metadata
I20260924 19:40:38.930372 17422 catalog_manager.cc:4841] TS b6f32728d20e43edacb85ac48dbbc374 (127.0.98.2:44951): tablet f6c051bc3c1843769ba56ca32d2fc562 (table test-table [id=24b21b43a9964cdb839b250c87e25e3b]) successfully deleted
I20260924 19:40:38.930586 17422 catalog_manager.cc:4841] TS 451d276cff484641aea55756f9d61a35 (127.0.98.3:40617): tablet f6c051bc3c1843769ba56ca32d2fc562 (table test-table [id=24b21b43a9964cdb839b250c87e25e3b]) successfully deleted
W20260924 19:40:38.932508 17423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44585 (error 108)
I20260924 19:40:38.932627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17633
I20260924 19:40:38.933604 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:38.933450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44585 (local address 127.0.0.1:43184)
0924 19:40:38.933516 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:38.933561 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 19:40:38.933732 17423 catalog_manager.cc:4584] TS 943e6ac46c7141838e19348cad5d01c5 (127.0.98.1:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c051bc3c1843769ba56ca32d2fc562: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44585: connect: Connection refused (error 111)
I20260924 19:40:38.937284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17767
I20260924 19:40:38.941526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17405
2026-09-24T19:40:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:38Z Disabled control of system clock
I20260924 19:40:38.967052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33693
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:39.049453 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:39.049664 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:39.049683 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:39.050832 17940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:39.050875 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:39.050886 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:39.050897 17940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:39.050916 17940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:39.052297 17940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:39.052525 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:39.052807 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:39.057497 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.057541 17945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.057605 17940 server_base.cc:1034] running on GCE node
W20260924 19:40:39.057549 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.057919 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:39.058218 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:39.059365 17940 hybrid_clock.cc:648] HybridClock initialized: now 1790278839059346 us; error 32 us; skew 500 ppm
I20260924 19:40:39.060675 17940 webserver.cc:460] Webserver started at http://127.0.98.62:38603/ using document root <none> and password file <none>
I20260924 19:40:39.060906 17940 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:39.060956 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:39.061082 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:39.062112 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "79eb431ff7e84455aa3175ff2db1cb37"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "bb65fc3439e9feb2c4e9786a5422976b"
server_key_iv: "d86c9a15d33a9be4e55780d0eea0bc02"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.062433 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "79eb431ff7e84455aa3175ff2db1cb37"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "bb65fc3439e9feb2c4e9786a5422976b"
server_key_iv: "d86c9a15d33a9be4e55780d0eea0bc02"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.063648 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:39.064363 17954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.064534 17940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.064638 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "79eb431ff7e84455aa3175ff2db1cb37"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "bb65fc3439e9feb2c4e9786a5422976b"
server_key_iv: "d86c9a15d33a9be4e55780d0eea0bc02"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.064710 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:39.073513 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:39.073793 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:39.073915 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:39.077453 17940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33693
I20260924 19:40:39.077500 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33693 every 8 connection(s)
I20260924 19:40:39.077800 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:39.078701 18007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:39.081023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 17940
I20260924 19:40:39.081032 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: Bootstrap starting.
I20260924 19:40:39.081130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:39.081313 392 external_mini_cluster.cc:1428] Setting key 914fd61e13c3d498eec352407e08bd41
I20260924 19:40:39.081669 18007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:39.082010 18007 log.cc:824] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:39.082715 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: No bootstrap required, opened a new log
I20260924 19:40:39.084992 18007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260924 19:40:39.085125 18007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:39.085147 18007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79eb431ff7e84455aa3175ff2db1cb37, State: Initialized, Role: FOLLOWER
I20260924 19:40:39.085306 18007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260924 19:40:39.085382 18007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:39.085417 18007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:39.085469 18007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:39.086127 18007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260924 19:40:39.086221 18007 leader_election.cc:302] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79eb431ff7e84455aa3175ff2db1cb37; no voters:
I20260924 19:40:39.086385 18007 leader_election.cc:288] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:39.086427 18012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:39.086580 18012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [term 1 LEADER]: Becoming Leader. State: Replica: 79eb431ff7e84455aa3175ff2db1cb37, State: Running, Role: LEADER
I20260924 19:40:39.086673 18007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:39.086730 18012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260924 19:40:39.087052 18012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79eb431ff7e84455aa3175ff2db1cb37. Latest consensus state: current_term: 1 leader_uuid: "79eb431ff7e84455aa3175ff2db1cb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } } }
I20260924 19:40:39.087116 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79eb431ff7e84455aa3175ff2db1cb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79eb431ff7e84455aa3175ff2db1cb37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } } }
I20260924 19:40:39.087255 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:39.087412 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.081782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.62:33693)
0924 19:40:39.082009 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.082018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.082051 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:40:39.082559 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.082568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.082643 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.082796 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:39.083027 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.084416 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.085192 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.085785 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.085908 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.086508 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:39.086512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:39.086512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:39.086705 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:39.086747 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:39.086761 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:39.086817 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.087025 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.087107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260924 19:40:39.087512 18021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:39.088048 18021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:39.087224 18012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:39.089644 18021 catalog_manager.cc:1347] Generated new cluster ID: 5da8d0921b6b490194f9b26e505358dd
I20260924 19:40:39.089697 18021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:39.099184 18021 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:39.099725 18021 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:39.102208 18021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: Generated new TSK 0
I20260924 19:40:39.102377 18021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:39.103776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--builtin_ntp_servers=127.0.98.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:39.190833 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:39.191041 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:39.191063 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:39.192263 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:39.192303 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:39.192328 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:39.193677 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:39.193903 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:39.194185 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:39.200044 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.200021 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.200029 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.200273 18031 server_base.cc:1034] running on GCE node
I20260924 19:40:39.200516 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:39.200778 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:39.201947 18031 hybrid_clock.cc:648] HybridClock initialized: now 1790278839201933 us; error 29 us; skew 500 ppm
I20260924 19:40:39.203155 18031 webserver.cc:460] Webserver started at http://127.0.98.1:41203/ using document root <none> and password file <none>
I20260924 19:40:39.203382 18031 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:39.203442 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:39.203573 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:39.204636 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "6df60de9990f4d0dba0c5f7c66ef339d"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "47ff21ed5c4adde452d333f72da2ba16"
server_key_iv: "e71b9ed19517fc818d1fbb36ab3d6811"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.204959 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6df60de9990f4d0dba0c5f7c66ef339d"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "47ff21ed5c4adde452d333f72da2ba16"
server_key_iv: "e71b9ed19517fc818d1fbb36ab3d6811"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.206240 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:39.207000 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.207158 18031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:39.207240 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "6df60de9990f4d0dba0c5f7c66ef339d"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "47ff21ed5c4adde452d333f72da2ba16"
server_key_iv: "e71b9ed19517fc818d1fbb36ab3d6811"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.207306 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:39.259341 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:39.259692 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:39.259828 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:39.260084 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:39.260406 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:39.260439 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.260470 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:39.260486 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.265736 18031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37871
I20260924 19:40:39.265825 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37871 every 8 connection(s)
I20260924 19:40:39.266098 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:39.269626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18031
I20260924 19:40:39.269726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:39.269912 392 external_mini_cluster.cc:1428] Setting key 6dd50bc77660f7ce78f919dd0788903c
I20260924 19:40:39.270788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--builtin_ntp_servers=127.0.98.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:39.271513 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.266777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36071 (local address 127.0.98.62:33693)
0924 19:40:39.266823 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.266825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.267108 (+ 283us) server_negotiation.cc:407] Connection header received
0924 19:40:39.267614 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.267614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.267623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.267680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:39.268379 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.268878 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.270097 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.270343 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.270401 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.271288 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:39.271289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:39.271289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:39.271293 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:39.271307 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:39.271311 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:39.271351 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.271436 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.271456 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:40:39.271752 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.266761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33693 (local address 127.0.98.1:36071)
0924 19:40:39.266956 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.267029 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.267531 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.267739 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.267742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.267750 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:39.268292 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.268298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.268947 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.268948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.269992 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.270005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.270424 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.270425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.270590 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.271134 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:39.271135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:39.271176 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:39.271205 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:39.271364 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:39.271368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:39.271371 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.271431 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.271539 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260924 19:40:39.272234 18159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33693
I20260924 19:40:39.272331 18159 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:39.272504 18159 heartbeater.cc:511] Master 127.0.98.62:33693 requested a full tablet report, sending...
I20260924 19:40:39.272998 17971 ts_manager.cc:194] Registered new tserver with Master: 6df60de9990f4d0dba0c5f7c66ef339d (127.0.98.1:37871)
I20260924 19:40:39.273838 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36071
W20260924 19:40:39.349772 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:39.349968 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:39.349988 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:39.351074 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:39.351107 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:39.351130 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:39.352437 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:39.352636 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:39.352872 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:39.357667 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.357713 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.357740 18162 server_base.cc:1034] running on GCE node
W20260924 19:40:39.357679 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.358062 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:39.358295 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:39.359434 18162 hybrid_clock.cc:648] HybridClock initialized: now 1790278839359424 us; error 27 us; skew 500 ppm
I20260924 19:40:39.360601 18162 webserver.cc:460] Webserver started at http://127.0.98.2:44265/ using document root <none> and password file <none>
I20260924 19:40:39.360783 18162 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:39.360821 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:39.360913 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:39.361887 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d06f79a360348b69961d6f00e2d97c4"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "790f3cb0aecaf716bdca61a7c5652d82"
server_key_iv: "2b9e83eb0b4fa37bb1699e067b8d4355"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.362178 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d06f79a360348b69961d6f00e2d97c4"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "790f3cb0aecaf716bdca61a7c5652d82"
server_key_iv: "2b9e83eb0b4fa37bb1699e067b8d4355"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.363255 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:39.363907 18176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.364125 18162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:39.364207 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "3d06f79a360348b69961d6f00e2d97c4"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "790f3cb0aecaf716bdca61a7c5652d82"
server_key_iv: "2b9e83eb0b4fa37bb1699e067b8d4355"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.364275 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:39.388017 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:39.388316 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:39.388447 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:39.388695 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:39.389007 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:39.389040 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.389061 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:39.389086 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.394237 18162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39145
I20260924 19:40:39.394330 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39145 every 8 connection(s)
I20260924 19:40:39.394584 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:39.396088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18162
I20260924 19:40:39.396189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:39.396384 392 external_mini_cluster.cc:1428] Setting key 5325169a84e0dd3c97e04b8def4f07a8
I20260924 19:40:39.397329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--builtin_ntp_servers=127.0.98.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:39.400357 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.395300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55611 (local address 127.0.98.62:33693)
0924 19:40:39.395355 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.395358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.395595 (+ 237us) server_negotiation.cc:407] Connection header received
0924 19:40:39.396267 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.396268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.396279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.396331 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:39.397163 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.397728 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.398973 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.399249 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.399315 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.400113 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:39.400114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:39.400114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:39.400119 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:39.400133 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:39.400137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:39.400176 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.400257 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.400281 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:40:39.400558 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.395279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33693 (local address 127.0.98.2:55611)
0924 19:40:39.395486 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.395525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.396116 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.396427 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.396432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.396444 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:39.397079 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.397086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.397813 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.397814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.398888 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.398890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.399323 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.399324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.399430 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.399971 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:39.399972 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:39.400001 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:39.400024 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:39.400184 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:39.400190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:39.400193 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.400255 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.400356 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260924 19:40:39.401021 18290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33693
I20260924 19:40:39.401115 18290 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:39.401283 18290 heartbeater.cc:511] Master 127.0.98.62:33693 requested a full tablet report, sending...
I20260924 19:40:39.401643 17971 ts_manager.cc:194] Registered new tserver with Master: 3d06f79a360348b69961d6f00e2d97c4 (127.0.98.2:39145)
I20260924 19:40:39.402050 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55611
W20260924 19:40:39.476100 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:39.476292 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:39.476311 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:39.477414 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:39.477456 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:39.477483 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:39.478814 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:39.479045 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:39.479317 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:39.484009 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.483987 18298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:39.484027 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:39.484186 18293 server_base.cc:1034] running on GCE node
I20260924 19:40:39.484418 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:39.484639 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:39.485809 18293 hybrid_clock.cc:648] HybridClock initialized: now 1790278839485795 us; error 26 us; skew 500 ppm
I20260924 19:40:39.487107 18293 webserver.cc:460] Webserver started at http://127.0.98.3:39095/ using document root <none> and password file <none>
I20260924 19:40:39.487335 18293 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:39.487385 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:39.487529 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:39.488672 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "b6cf7220797e17d9d85ad74ca5addb80"
server_key_iv: "497bf692e601b8938bf69235edebb955"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.489006 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "b6cf7220797e17d9d85ad74ca5addb80"
server_key_iv: "497bf692e601b8938bf69235edebb955"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.490216 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:39.490883 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.491034 18293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:39.491110 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf"
format_stamp: "Formatted at 2026-09-24 19:40:39 on dist-test-slave-6kf3"
server_key: "b6cf7220797e17d9d85ad74ca5addb80"
server_key_iv: "497bf692e601b8938bf69235edebb955"
server_key_version: "encryptionkey@0"
I20260924 19:40:39.491178 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:39.531050 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:39.531374 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:39.531512 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:39.531745 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:39.532102 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:39.532136 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.532166 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:39.532183 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:39.537299 18293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45317
I20260924 19:40:39.537362 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45317 every 8 connection(s)
I20260924 19:40:39.537652 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:40:39.542594 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.538354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36385 (local address 127.0.98.62:33693)
0924 19:40:39.538433 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.538435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.538685 (+ 250us) server_negotiation.cc:407] Connection header received
0924 19:40:39.539167 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.539168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.539178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.539229 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:39.539868 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.540387 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.541305 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.541492 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.541541 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.542358 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:39.542359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:39.542359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:39.542363 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:39.542379 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:39.542384 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:39.542428 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.542504 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.542524 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:40:39.542735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18293
I20260924 19:40:39.542819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:39.542858 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.538329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33693 (local address 127.0.98.3:36385)
0924 19:40:39.538544 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.538599 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.539088 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.539283 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.539286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.539294 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:39.539775 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.539779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.540460 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.540460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.541235 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.541237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.541553 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.541554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.541673 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.542213 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:39.542214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:39.542242 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:39.542264 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:39.542438 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:39.542440 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:39.542443 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.542489 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.542597 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260924 19:40:39.543018 392 external_mini_cluster.cc:1428] Setting key 9ce5580a53543df3f270fd668f87f1aa
I20260924 19:40:39.543545 18421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33693
I20260924 19:40:39.543634 18421 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:39.543798 18421 heartbeater.cc:511] Master 127.0.98.62:33693 requested a full tablet report, sending...
I20260924 19:40:39.544130 17971 ts_manager.cc:194] Registered new tserver with Master: 34f9ca98c14d4d9383bfcd0fa47a8baf (127.0.98.3:45317)
I20260924 19:40:39.544476 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36385
I20260924 19:40:39.554598 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:39.558404 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.555844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.0.98.62:33693)
0924 19:40:39.555890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.555892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.555999 (+ 107us) server_negotiation.cc:407] Connection header received
0924 19:40:39.556052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.556053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.556061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.556120 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:39.556255 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.556718 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.557449 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.557637 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.557696 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.558221 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:39.558222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:39.558223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:39.558226 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:39.558238 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:39.558242 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:39.558279 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.558329 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.558348 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:40:39.560284 17971 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:39.561875 17971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:39.568136 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.98.3:45317)
0924 19:40:39.564431 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.564435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.564446 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:40:39.564483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.564485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.564499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.564590 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:39.564808 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565870 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.566893 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567701 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567803 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.567961 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.568010 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.568050 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":93,"threads_started":1}
I20260924 19:40:39.568193 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.563849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45317 (local address 127.0.0.1:37132)
0924 19:40:39.564045 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.564086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.564152 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.564671 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.564671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.564673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:39.564711 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.564712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565965 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.565966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.566826 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.566829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567815 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.567892 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.567907 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.567996 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.568058 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260924 19:40:39.568282 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.564214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.0.98.2:39145)
0924 19:40:39.564434 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.564437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.564446 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:39.564483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.564485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.564499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.564580 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:39.564854 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565560 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.566977 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567705 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567771 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.567870 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.568162 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.568201 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260924 19:40:39.568367 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.564115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.98.1:37871)
0924 19:40:39.564338 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.564343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.564352 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:39.564389 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.564391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.564406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.564478 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:39.564689 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565846 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567254 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567786 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.568130 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.568223 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.568249 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.568289 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260924 19:40:39.568387 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.563847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:41070)
0924 19:40:39.564045 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.564086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.564151 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.564493 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.564496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.564498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:39.564628 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.564631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565933 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.565934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.567173 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.567175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567905 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.567986 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.567989 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.568016 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.568058 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":406,"threads_started":1}
I20260924 19:40:39.568607 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.563847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39145 (local address 127.0.0.1:34664)
0924 19:40:39.564290 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.564299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.564338 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.564776 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.564776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.564778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:39.564801 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.564803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.565640 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.565641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.566897 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.566900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.567903 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.567904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.567988 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.567992 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.568039 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.568087 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":2,"thread_start_us":123,"threads_started":1}
I20260924 19:40:39.569236 18355 tablet_service.cc:1467] Processing CreateTablet for tablet c04f7fe80df444c8ae36a208ac62e9de (DEFAULT_TABLE table=test-table [id=fd6919f88b254c3aa0261870d13d76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:39.569224 18093 tablet_service.cc:1467] Processing CreateTablet for tablet c04f7fe80df444c8ae36a208ac62e9de (DEFAULT_TABLE table=test-table [id=fd6919f88b254c3aa0261870d13d76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:39.569223 18224 tablet_service.cc:1467] Processing CreateTablet for tablet c04f7fe80df444c8ae36a208ac62e9de (DEFAULT_TABLE table=test-table [id=fd6919f88b254c3aa0261870d13d76ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:39.569561 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04f7fe80df444c8ae36a208ac62e9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:39.569561 18093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04f7fe80df444c8ae36a208ac62e9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:39.569571 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04f7fe80df444c8ae36a208ac62e9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:39.572013 18435 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Bootstrap starting.
I20260924 19:40:39.572641 18435 tablet_bootstrap.cc:654] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:39.572656 18437 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Bootstrap starting.
I20260924 19:40:39.573040 18435 log.cc:824] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:39.573271 18437 tablet_bootstrap.cc:654] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:39.573556 18437 log.cc:824] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:39.573753 18435 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: No bootstrap required, opened a new log
I20260924 19:40:39.573822 18435 ts_tablet_manager.cc:1397] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:39.573868 18436 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Bootstrap starting.
I20260924 19:40:39.574453 18436 tablet_bootstrap.cc:654] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:39.574761 18436 log.cc:824] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:39.575282 18437 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: No bootstrap required, opened a new log
I20260924 19:40:39.575387 18437 ts_tablet_manager.cc:1397] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:40:39.575598 18436 tablet_bootstrap.cc:492] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: No bootstrap required, opened a new log
I20260924 19:40:39.575693 18436 ts_tablet_manager.cc:1397] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:40:39.576398 18435 raft_consensus.cc:348] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.576581 18435 raft_consensus.cc:374] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:39.576615 18435 raft_consensus.cc:729] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f9ca98c14d4d9383bfcd0fa47a8baf, State: Initialized, Role: FOLLOWER
I20260924 19:40:39.576763 18435 consensus_queue.cc:260] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.577006 18435 ts_tablet_manager.cc:1428] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:40:39.577153 18421 heartbeater.cc:503] Master 127.0.98.62:33693 was elected leader, sending a full tablet report...
I20260924 19:40:39.577649 18437 raft_consensus.cc:348] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.577752 18437 raft_consensus.cc:374] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:39.577770 18437 raft_consensus.cc:729] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d06f79a360348b69961d6f00e2d97c4, State: Initialized, Role: FOLLOWER
I20260924 19:40:39.577852 18437 consensus_queue.cc:260] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.577864 18436 raft_consensus.cc:348] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.578034 18436 raft_consensus.cc:374] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:39.578071 18437 ts_tablet_manager.cc:1428] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:40:39.578091 18436 raft_consensus.cc:729] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df60de9990f4d0dba0c5f7c66ef339d, State: Initialized, Role: FOLLOWER
I20260924 19:40:39.578192 18290 heartbeater.cc:503] Master 127.0.98.62:33693 was elected leader, sending a full tablet report...
I20260924 19:40:39.578233 18436 consensus_queue.cc:260] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.578466 18436 ts_tablet_manager.cc:1428] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:40:39.578588 18159 heartbeater.cc:503] Master 127.0.98.62:33693 was elected leader, sending a full tablet report...
W20260924 19:40:39.645138 18291 tablet.cc:2369] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:39.766777 18160 tablet.cc:2369] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:39.788177 18422 tablet.cc:2369] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:39.885589 18443 raft_consensus.cc:482] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:39.885705 18443 raft_consensus.cc:504] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.886052 18443 leader_election.cc:288] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d06f79a360348b69961d6f00e2d97c4 (127.0.98.2:39145), 34f9ca98c14d4d9383bfcd0fa47a8baf (127.0.98.3:45317)
I20260924 19:40:39.889076 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.886307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44369 (local address 127.0.98.3:45317)
0924 19:40:39.886360 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.886362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.886391 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:40:39.886461 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.886462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.886472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.886522 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:39.886687 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.887239 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888409 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.888770 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888830 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.888839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.888977 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.889010 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:40:39.889112 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.886259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48789 (local address 127.0.98.2:39145)
0924 19:40:39.886305 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.886307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.886427 (+ 120us) server_negotiation.cc:407] Connection header received
0924 19:40:39.886469 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.886470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.886479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.886529 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:39.886727 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.887261 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888414 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.888752 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888805 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.888814 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.888972 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.889009 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:40:39.889106 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.886207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45317 (local address 127.0.98.1:44369)
0924 19:40:39.886335 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.886345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.886402 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.886536 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.886537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.886539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:39.886610 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.886612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.887309 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.887310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.888329 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.888332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.888846 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.888914 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.888917 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.888967 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.889005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":39,"thread_start_us":57,"threads_started":1}
I20260924 19:40:39.889112 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.886207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39145 (local address 127.0.98.1:48789)
0924 19:40:39.886346 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.886355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:39.886419 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:39.886542 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:39.886543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:39.886545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:39.886636 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.886638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.887336 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.887337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.888327 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:39.888329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.888830 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.888831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:39.888899 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.888903 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:39.888944 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:40:39.888985 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":34,"thread_start_us":63,"threads_started":1}
I20260924 19:40:39.889438 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04f7fe80df444c8ae36a208ac62e9de" candidate_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" is_pre_election: true
I20260924 19:40:39.889593 18375 raft_consensus.cc:2454] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df60de9990f4d0dba0c5f7c66ef339d in term 0.
I20260924 19:40:39.889569 18244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04f7fe80df444c8ae36a208ac62e9de" candidate_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d06f79a360348b69961d6f00e2d97c4" is_pre_election: true
I20260924 19:40:39.889729 18244 raft_consensus.cc:2454] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df60de9990f4d0dba0c5f7c66ef339d in term 0.
I20260924 19:40:39.889822 18049 leader_election.cc:302] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f9ca98c14d4d9383bfcd0fa47a8baf, 6df60de9990f4d0dba0c5f7c66ef339d; no voters:
I20260924 19:40:39.889940 18443 raft_consensus.cc:2781] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:39.889991 18443 raft_consensus.cc:482] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:39.890007 18443 raft_consensus.cc:3037] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:39.890713 18443 raft_consensus.cc:504] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.890832 18443 leader_election.cc:288] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [CANDIDATE]: Term 1 election: Requested vote from peers 3d06f79a360348b69961d6f00e2d97c4 (127.0.98.2:39145), 34f9ca98c14d4d9383bfcd0fa47a8baf (127.0.98.3:45317)
I20260924 19:40:39.890971 18244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04f7fe80df444c8ae36a208ac62e9de" candidate_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d06f79a360348b69961d6f00e2d97c4"
I20260924 19:40:39.891036 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04f7fe80df444c8ae36a208ac62e9de" candidate_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf"
I20260924 19:40:39.891038 18244 raft_consensus.cc:3037] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:39.891117 18375 raft_consensus.cc:3037] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:39.891748 18244 raft_consensus.cc:2454] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df60de9990f4d0dba0c5f7c66ef339d in term 1.
I20260924 19:40:39.891834 18375 raft_consensus.cc:2454] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df60de9990f4d0dba0c5f7c66ef339d in term 1.
I20260924 19:40:39.891916 18048 leader_election.cc:302] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d06f79a360348b69961d6f00e2d97c4, 6df60de9990f4d0dba0c5f7c66ef339d; no voters:
I20260924 19:40:39.892035 18443 raft_consensus.cc:2781] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:39.892169 18443 raft_consensus.cc:686] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 1 LEADER]: Becoming Leader. State: Replica: 6df60de9990f4d0dba0c5f7c66ef339d, State: Running, Role: LEADER
I20260924 19:40:39.892263 18443 consensus_queue.cc:237] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260924 19:40:39.893028 17971 catalog_manager.cc:5495] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6df60de9990f4d0dba0c5f7c66ef339d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6df60de9990f4d0dba0c5f7c66ef339d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260924 19:40:39.923517 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.921277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.98.1:37871)
0924 19:40:39.921335 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:39.921337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:39.921346 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:39.921433 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:39.921434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:39.921451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:39.921508 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:39.921621 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.922084 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.922813 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:39.922998 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:39.923051 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:39.923211 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:39.923318 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:39.923389 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:39.923424 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 19:40:39.923440 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:40:39.925228 18375 raft_consensus.cc:1261] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 1 FOLLOWER]: Refusing update from remote peer 6df60de9990f4d0dba0c5f7c66ef339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:39.925226 18244 raft_consensus.cc:1261] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 1 FOLLOWER]: Refusing update from remote peer 6df60de9990f4d0dba0c5f7c66ef339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:39.925527 18443 consensus_queue.cc:1035] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d06f79a360348b69961d6f00e2d97c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:39.925683 18446 consensus_queue.cc:1035] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f9ca98c14d4d9383bfcd0fa47a8baf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:39.927708 18449 mvcc.cc:205] Tried to move back new op lower bound from 7332982128331272192 to 7332982128199270400. Current Snapshot: MvccSnapshot[applied={T|T < 7332982128331272192}]
I20260924 19:40:39.928296 18451 mvcc.cc:205] Tried to move back new op lower bound from 7332982128331272192 to 7332982128199270400. Current Snapshot: MvccSnapshot[applied={T|T < 7332982128331272192}]
I20260924 19:40:39.930521 17969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:39.930603 17969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:39.931275 17969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79eb431ff7e84455aa3175ff2db1cb37: Sending DeleteTablet for 3 replicas of tablet c04f7fe80df444c8ae36a208ac62e9de
I20260924 19:40:39.931600 18093 tablet_service.cc:1514] Processing DeleteTablet for tablet c04f7fe80df444c8ae36a208ac62e9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:39 UTC) from {username='slave'} at 127.0.0.1:41070
I20260924 19:40:39.931596 18224 tablet_service.cc:1514] Processing DeleteTablet for tablet c04f7fe80df444c8ae36a208ac62e9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:39 UTC) from {username='slave'} at 127.0.0.1:34664
I20260924 19:40:39.931803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18031
I20260924 19:40:39.931857 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet c04f7fe80df444c8ae36a208ac62e9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:39 UTC) from {username='slave'} at 127.0.0.1:37132
I20260924 19:40:39.932010 18460 tablet_replica.cc:330] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d: stopping tablet replica
I20260924 19:40:39.932096 18460 raft_consensus.cc:2229] T c04f7fe80df444c8ae36a208ac62e9de P 6df60de9990f4d0dba0c5f7c66ef339d [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:39.932359 18459 tablet_replica.cc:330] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: stopping tablet replica
I20260924 19:40:39.932438 18459 raft_consensus.cc:2229] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:39.932526 18459 raft_consensus.cc:2258] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:39.932932 18459 ts_tablet_manager.cc:1905] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:39.934435 18459 ts_tablet_manager.cc:1918] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:39.934506 18459 log.cc:1196] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/c04f7fe80df444c8ae36a208ac62e9de
I20260924 19:40:39.934734 18459 ts_tablet_manager.cc:1939] T c04f7fe80df444c8ae36a208ac62e9de P 3d06f79a360348b69961d6f00e2d97c4: Deleting consensus metadata
I20260924 19:40:39.935110 17957 catalog_manager.cc:4841] TS 3d06f79a360348b69961d6f00e2d97c4 (127.0.98.2:39145): tablet c04f7fe80df444c8ae36a208ac62e9de (table test-table [id=fd6919f88b254c3aa0261870d13d76ee]) successfully deleted
I20260924 19:40:39.936056 18461 tablet_replica.cc:330] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: stopping tablet replica
I20260924 19:40:39.936159 18461 raft_consensus.cc:2229] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:39.936259 18461 raft_consensus.cc:2258] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:39.936616 18461 ts_tablet_manager.cc:1905] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:40:39.937318 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37871 (error 108)
I20260924 19:40:39.937721 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:39.937606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:41086)
0924 19:40:39.937653 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:39.937690 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:40:39.937815 17956 catalog_manager.cc:4584] TS 6df60de9990f4d0dba0c5f7c66ef339d (127.0.98.1:37871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c04f7fe80df444c8ae36a208ac62e9de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
I20260924 19:40:39.937938 18461 ts_tablet_manager.cc:1918] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:39.938010 18461 log.cc:1196] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/c04f7fe80df444c8ae36a208ac62e9de
I20260924 19:40:39.938076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18162
I20260924 19:40:39.938722 18461 ts_tablet_manager.cc:1939] T c04f7fe80df444c8ae36a208ac62e9de P 34f9ca98c14d4d9383bfcd0fa47a8baf: Deleting consensus metadata
I20260924 19:40:39.939080 17958 catalog_manager.cc:4841] TS 34f9ca98c14d4d9383bfcd0fa47a8baf (127.0.98.3:45317): tablet c04f7fe80df444c8ae36a208ac62e9de (table test-table [id=fd6919f88b254c3aa0261870d13d76ee]) successfully deleted
I20260924 19:40:39.942255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18293
I20260924 19:40:39.946398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 17940
2026-09-24T19:40:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:39 dist-test-slave-6kf3 krb5kdc[18465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:39 dist-test-slave-6kf3 krb5kdc[18465](info): set up 2 sockets
Sep 24 19:40:39 dist-test-slave-6kf3 krb5kdc[18465](info): commencing operation
krb5kdc: starting...
W20260924 19:40:40.895434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:40 dist-test-slave-6kf3 krb5kdc[18465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:40:41.007109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:41.087145 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:41.087357 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:41.087388 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:41.088555 18478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:41.088598 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:41.088617 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:41.088634 18478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:41.088649 18478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:41.090099 18478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:41.090337 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:41.090544 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:41.095199 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.095211 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.095360 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.095440 18478 server_base.cc:1034] running on GCE node
I20260924 19:40:41.095729 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:41.096056 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:41.097237 18478 hybrid_clock.cc:648] HybridClock initialized: now 1790278841097211 us; error 38 us; skew 500 ppm
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:41.104585 18478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:40:41.105072 18478 webserver.cc:460] Webserver started at http://127.0.98.62:36401/ using document root <none> and password file <none>
I20260924 19:40:41.105301 18478 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:41.105353 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:41.105484 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:41.106531 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "6178f1372a274c8a82b259ca7841f3e3"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "89cacc756e4541e20daf88e859494881"
server_key_iv: "682f8ad17977d23f8eaee49b4d868347"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.106854 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "6178f1372a274c8a82b259ca7841f3e3"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "89cacc756e4541e20daf88e859494881"
server_key_iv: "682f8ad17977d23f8eaee49b4d868347"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.108155 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:41.108819 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.108974 18478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:41.109088 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "6178f1372a274c8a82b259ca7841f3e3"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "89cacc756e4541e20daf88e859494881"
server_key_iv: "682f8ad17977d23f8eaee49b4d868347"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.109153 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:41.125838 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:41.127696 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:41.127837 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:41.131554 18478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34455
I20260924 19:40:41.131608 18545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34455 every 8 connection(s)
I20260924 19:40:41.132015 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:41.132874 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:41.135231 18546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: Bootstrap starting.
I20260924 19:40:41.135818 18546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:41.136135 18546 log.cc:824] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:41.136873 18546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: No bootstrap required, opened a new log
I20260924 19:40:41.138441 18546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 19:40:41.138581 18546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:41.138602 18546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6178f1372a274c8a82b259ca7841f3e3, State: Initialized, Role: FOLLOWER
I20260924 19:40:41.138767 18546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 19:40:41.138832 18546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:41.138852 18546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:41.138880 18546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:41.139360 18546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 19:40:41.139456 18546 leader_election.cc:302] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6178f1372a274c8a82b259ca7841f3e3; no voters:
I20260924 19:40:41.139619 18546 leader_election.cc:288] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:41.139671 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:41.139838 18546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:41.139851 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [term 1 LEADER]: Becoming Leader. State: Replica: 6178f1372a274c8a82b259ca7841f3e3, State: Running, Role: LEADER
I20260924 19:40:41.139999 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 19:40:41.140388 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6178f1372a274c8a82b259ca7841f3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260924 19:40:41.140468 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:41.140534 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6178f1372a274c8a82b259ca7841f3e3. Latest consensus state: current_term: 1 leader_uuid: "6178f1372a274c8a82b259ca7841f3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6178f1372a274c8a82b259ca7841f3e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260924 19:40:41.140611 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:41.140666 18558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:41.141191 18558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:41.141793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18478
I20260924 19:40:41.141889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:41.142071 392 external_mini_cluster.cc:1428] Setting key a3e0e65f446f6bc82785a2c2736362ab
I20260924 19:40:41.142756 18558 catalog_manager.cc:1347] Generated new cluster ID: 8b41356913ec459994bd3e02b9162b1f
I20260924 19:40:41.142803 18558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:41.151000 18558 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:41.151425 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.142698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.0.98.62:34455)
0924 19:40:41.142847 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.142857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.142888 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:40:41.142985 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.142991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.143032 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.143147 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:41.143784 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.144842 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.145580 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.145993 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.146086 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.148285 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:41.148289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:41.148289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:41.148301 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:41.149796 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.150159 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.150163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.150164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.150275 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.150494 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.150495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.150496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.150787 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:41.150959 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.151122 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.151265 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 19:40:41.151714 18558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:41.154314 18558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: Generated new TSK 0
I20260924 19:40:41.154469 18558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:40:41.187160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:41.267258 18572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:41.267455 18572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:41.267485 18572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:41.268680 18572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:41.268725 18572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:41.268759 18572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:41.270208 18572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:41.270437 18572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:41.270644 18572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:41.275480 18577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.275416 18578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.275413 18572 server_base.cc:1034] running on GCE node
W20260924 19:40:41.275525 18580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.275848 18572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:41.276116 18572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:41.277252 18572 hybrid_clock.cc:648] HybridClock initialized: now 1790278841277244 us; error 29 us; skew 500 ppm
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:41.279660 18572 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:40:41.280076 18572 webserver.cc:460] Webserver started at http://127.0.98.1:34679/ using document root <none> and password file <none>
I20260924 19:40:41.280273 18572 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:41.280313 18572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:41.280403 18572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:41.281379 18572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1560ac3297f4279b5250ce138caebe7"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "07ef85f4dcc213f1ed5090e53e4a80ac"
server_key_iv: "863b9da32a04daa4fab847233348a429"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.281651 18572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1560ac3297f4279b5250ce138caebe7"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "07ef85f4dcc213f1ed5090e53e4a80ac"
server_key_iv: "863b9da32a04daa4fab847233348a429"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.282780 18572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:40:41.283468 18587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.283603 18572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:40:41.283679 18572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "e1560ac3297f4279b5250ce138caebe7"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "07ef85f4dcc213f1ed5090e53e4a80ac"
server_key_iv: "863b9da32a04daa4fab847233348a429"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.283748 18572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:41.292594 18572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:41.294514 18572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:41.294646 18572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:41.294844 18572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:41.295120 18572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:41.295150 18572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.295171 18572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:41.295193 18572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.300568 18572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45619
I20260924 19:40:41.300633 18700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45619 every 8 connection(s)
I20260924 19:40:41.300887 18572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:41.302429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18572
I20260924 19:40:41.302531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:41.302742 392 external_mini_cluster.cc:1428] Setting key 2dc5afdef6e839dbc77abacf1460aa86
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:40:41.310189 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.301518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43849 (local address 127.0.98.62:34455)
0924 19:40:41.301585 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.301587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.301767 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:40:41.302687 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.302689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.302702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.302776 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:41.303480 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.304209 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.305182 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.305428 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.305484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.307682 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:41.307684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:41.307684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:41.307691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:41.308956 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.309310 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.309311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.309312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.309360 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.309571 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.309572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.309572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.309678 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:41.309732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.310002 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.310099 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:40:41.310341 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.301480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34455 (local address 127.0.98.1:43849)
0924 19:40:41.301648 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.301690 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.302573 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.302857 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.302861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.303160 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:41.303392 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.303398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.304286 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.304287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.305096 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.305097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.305501 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.305502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.305639 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.306092 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:41.306100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:41.307575 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:41.309032 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.309040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.309042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.309243 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.309415 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.309416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.309416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.309515 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.309736 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:41.309742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:41.309880 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.309982 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.310129 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260924 19:40:41.310820 18701 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34455
I20260924 19:40:41.310909 18701 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:41.311095 18701 heartbeater.cc:511] Master 127.0.98.62:34455 requested a full tablet report, sending...
I20260924 19:40:41.311633 18510 ts_manager.cc:194] Registered new tserver with Master: e1560ac3297f4279b5250ce138caebe7 (127.0.98.1:45619)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 19:40:41.312393 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:40:41.324478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:41.402401 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:41.402603 18706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:41.402635 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:41.403718 18706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:41.403750 18706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:41.403774 18706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:41.405265 18706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:41.405447 18706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:41.405627 18706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:41.410413 18711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.410420 18714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.410429 18712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.410717 18706 server_base.cc:1034] running on GCE node
I20260924 19:40:41.410912 18706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:41.411131 18706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:41.412268 18706 hybrid_clock.cc:648] HybridClock initialized: now 1790278841412254 us; error 28 us; skew 500 ppm
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:41.414664 18706 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:40:41.415069 18706 webserver.cc:460] Webserver started at http://127.0.98.2:33379/ using document root <none> and password file <none>
I20260924 19:40:41.415242 18706 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:41.415282 18706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:41.415369 18706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:41.416378 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "7777782c9f6f49e89ddde8461aa41b64"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "3117d124b2e475a9161b6dd29fb085ae"
server_key_iv: "c9c41a863ebe047ecb53080cf490239b"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.416679 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7777782c9f6f49e89ddde8461aa41b64"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "3117d124b2e475a9161b6dd29fb085ae"
server_key_iv: "c9c41a863ebe047ecb53080cf490239b"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.417806 18706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:41.418434 18721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.418601 18706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:41.418695 18706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "7777782c9f6f49e89ddde8461aa41b64"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "3117d124b2e475a9161b6dd29fb085ae"
server_key_iv: "c9c41a863ebe047ecb53080cf490239b"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.418751 18706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:41.434038 18706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:41.436021 18706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:41.436141 18706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:41.436319 18706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:41.436554 18706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:41.436582 18706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.436604 18706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:41.436615 18706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.441937 18706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43699
I20260924 19:40:41.441978 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43699 every 8 connection(s)
I20260924 19:40:41.442255 18706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:41.448762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18706
I20260924 19:40:41.448851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:41.449038 392 external_mini_cluster.cc:1428] Setting key 1b3dfb0e98ce5f833c3147f8b59aaf84
I20260924 19:40:41.450659 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.442917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39713 (local address 127.0.98.62:34455)
0924 19:40:41.442976 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.442978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.443167 (+ 189us) server_negotiation.cc:407] Connection header received
0924 19:40:41.444071 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.444072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.444082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.444136 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:41.444671 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.445158 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.446092 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.446272 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.446318 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.448173 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:41.448175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:41.448175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:41.448181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:41.449423 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.449817 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.449818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.449818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.449867 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.450111 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.450111 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.450111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.450202 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:41.450247 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.450523 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.450598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:40:41.450881 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.442881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34455 (local address 127.0.98.2:39713)
0924 19:40:41.443048 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.443087 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.443984 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.444191 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.444195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.444419 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:41.444602 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.444606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.445229 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.445229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.446021 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.446024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.446333 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.446334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.446428 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.446807 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:41.446814 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:41.448094 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:41.449497 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.449502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.449503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.449750 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.449922 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.449923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.449923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.450049 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.450256 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:41.450261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:41.450411 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.450532 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.450677 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 19:40:41.451346 18835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34455
I20260924 19:40:41.451440 18835 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:41.451608 18835 heartbeater.cc:511] Master 127.0.98.62:34455 requested a full tablet report, sending...
I20260924 19:40:41.451923 18510 ts_manager.cc:194] Registered new tserver with Master: 7777782c9f6f49e89ddde8461aa41b64 (127.0.98.2:43699)
I20260924 19:40:41.452406 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39713
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:40:41.470760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:41.548775 18840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:41.548964 18840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:41.548986 18840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:41.550093 18840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:41.550132 18840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:41.550156 18840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:41.551617 18840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:41.551820 18840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:41.552045 18840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:41.556628 18845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.556656 18846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.556797 18840 server_base.cc:1034] running on GCE node
W20260924 19:40:41.556839 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.557106 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:41.557296 18840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:41.558441 18840 hybrid_clock.cc:648] HybridClock initialized: now 1790278841558427 us; error 25 us; skew 500 ppm
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:41.560859 18840 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:40:41.561285 18840 webserver.cc:460] Webserver started at http://127.0.98.3:42859/ using document root <none> and password file <none>
I20260924 19:40:41.561475 18840 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:41.561522 18840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:41.561650 18840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:41.562635 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "eddb0442c95d4cb78236ed426d469a4e"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "4814f48f6576b9333a64408629737662"
server_key_iv: "11a212f8721975c9ce249a14294e2ba8"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.562986 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eddb0442c95d4cb78236ed426d469a4e"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "4814f48f6576b9333a64408629737662"
server_key_iv: "11a212f8721975c9ce249a14294e2ba8"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.564296 18840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:41.564944 18855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.565133 18840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:41.565210 18840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "eddb0442c95d4cb78236ed426d469a4e"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "4814f48f6576b9333a64408629737662"
server_key_iv: "11a212f8721975c9ce249a14294e2ba8"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.565258 18840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:41.572729 18840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:41.574653 18840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:41.574769 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:41.574956 18840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:41.575237 18840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:41.575268 18840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.575294 18840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:41.575310 18840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.580647 18840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38881
I20260924 19:40:41.580731 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38881 every 8 connection(s)
I20260924 19:40:41.580982 18840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:40:41.585042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 18840
I20260924 19:40:41.585150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:41.585325 392 external_mini_cluster.cc:1428] Setting key 623edea54f5c9319104e6aac03595c48
Sep 24 19:40:41 dist-test-slave-6kf3 krb5kdc[18465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:41.589525 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.581659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44345 (local address 127.0.98.62:34455)
0924 19:40:41.581739 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.581741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.581921 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:40:41.582783 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.582784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.582796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.582863 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:41.583449 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.583992 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.584944 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.585129 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.585209 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.587286 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:41.587288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:41.587288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:41.587295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:41.588469 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.588823 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.588824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.588824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.588875 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.589043 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.589044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.589044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.589140 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:41.589185 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.589390 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.589445 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:40:41.589655 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.581611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34455 (local address 127.0.98.3:44345)
0924 19:40:41.581800 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.581838 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.582682 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.582934 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.582937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.583172 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:41.583355 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.583359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.584068 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.584069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.584866 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.584868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.585219 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.585220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.585356 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.585894 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:41.585902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:41.587204 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:41.588557 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.588563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.588564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.588761 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.588915 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:41.588916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:41.588916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:41.588995 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:41.589186 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:41.589189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:41.589293 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.589368 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.589476 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260924 19:40:41.590094 18969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34455
I20260924 19:40:41.590180 18969 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:41.590339 18969 heartbeater.cc:511] Master 127.0.98.62:34455 requested a full tablet report, sending...
I20260924 19:40:41.590626 18510 ts_manager.cc:194] Registered new tserver with Master: eddb0442c95d4cb78236ed426d469a4e (127.0.98.3:38881)
I20260924 19:40:41.590966 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44345
I20260924 19:40:41.596943 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:41.604058 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.598219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.0.98.62:34455)
0924 19:40:41.598257 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.598259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.598389 (+ 130us) server_negotiation.cc:407] Connection header received
0924 19:40:41.598446 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.598447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.598456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.598500 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:41.599073 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.599553 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.600328 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.600542 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.600596 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.601816 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:41.601818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:41.601818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:41.601825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:41.602834 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.603251 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.603252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.603252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.603304 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:41.603495 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:41.603496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:41.603496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:41.603643 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:41.603736 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.603877 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.603964 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:40:41.605650 18510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:41.607228 18510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:41.613216 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38881 (local address 127.0.0.1:39922)
0924 19:40:41.609546 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.609600 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.609661 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.610072 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.610073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.610075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:41.610121 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.610123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.610933 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.610934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.612127 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.612130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.612922 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.612923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.612998 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.613012 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.613076 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.613139 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260924 19:40:41.613226 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.98.3:38881)
0924 19:40:41.609945 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.609949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.609958 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:41.609992 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.609994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.610007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.610073 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:41.610165 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.610839 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.612220 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.612849 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.612912 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.612973 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.613102 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.613139 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 19:40:41.613505 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43699 (local address 127.0.0.1:54286)
0924 19:40:41.609744 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.609752 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.609786 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.609994 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.609998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.610000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:41.610089 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.610091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.611314 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.611315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.612384 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.612386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.613288 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.613289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.613359 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.613363 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.613393 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.613432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260924 19:40:41.613792 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45619 (local address 127.0.0.1:58672)
0924 19:40:41.609546 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.609595 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.609661 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.610430 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.610431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.610433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:41.610466 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.610468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.611703 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.611704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.612726 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.612729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.613572 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.613573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.613641 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.613645 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.613698 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.613732 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260924 19:40:41.613818 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.98.2:43699)
0924 19:40:41.609814 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.609819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.609829 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:40:41.609866 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.609867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.609884 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.609959 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:41.610170 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.611206 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.612462 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.613201 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.613622 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.613685 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.613708 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.613744 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260924 19:40:41.613819 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.609540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.1:45619)
0924 19:40:41.609954 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.609958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.609969 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:40:41.610002 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.610004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.610018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.610552 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:41.610628 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.611622 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.612807 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.613493 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.613557 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.613613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.613708 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.613743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":74,"threads_started":1}
I20260924 19:40:41.614171 18903 tablet_service.cc:1467] Processing CreateTablet for tablet 14cff128b9614e0a99f783523dd44796 (DEFAULT_TABLE table=test-table [id=1e47446ff616430ca23102631a58312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:41.614432 18903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cff128b9614e0a99f783523dd44796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:41.614353 18635 tablet_service.cc:1467] Processing CreateTablet for tablet 14cff128b9614e0a99f783523dd44796 (DEFAULT_TABLE table=test-table [id=1e47446ff616430ca23102631a58312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:41.614627 18635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cff128b9614e0a99f783523dd44796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:41.616954 18983 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Bootstrap starting.
I20260924 19:40:41.616999 18984 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Bootstrap starting.
I20260924 19:40:41.617525 18984 tablet_bootstrap.cc:654] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:41.617578 18983 tablet_bootstrap.cc:654] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:41.617547 18769 tablet_service.cc:1467] Processing CreateTablet for tablet 14cff128b9614e0a99f783523dd44796 (DEFAULT_TABLE table=test-table [id=1e47446ff616430ca23102631a58312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:41.617858 18769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cff128b9614e0a99f783523dd44796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:41.617905 18984 log.cc:824] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:41.618084 18983 log.cc:824] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:41.618486 18984 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: No bootstrap required, opened a new log
I20260924 19:40:41.618530 18984 ts_tablet_manager.cc:1397] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:41.620057 18984 raft_consensus.cc:348] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.620234 18984 raft_consensus.cc:374] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:41.620289 18984 raft_consensus.cc:729] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1560ac3297f4279b5250ce138caebe7, State: Initialized, Role: FOLLOWER
I20260924 19:40:41.620425 18984 consensus_queue.cc:260] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.620728 18984 ts_tablet_manager.cc:1428] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:40:41.620806 18701 heartbeater.cc:503] Master 127.0.98.62:34455 was elected leader, sending a full tablet report...
I20260924 19:40:41.620951 18983 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: No bootstrap required, opened a new log
I20260924 19:40:41.621009 18983 ts_tablet_manager.cc:1397] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:40:41.622545 18983 raft_consensus.cc:348] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.622697 18983 raft_consensus.cc:374] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:41.622733 18983 raft_consensus.cc:729] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddb0442c95d4cb78236ed426d469a4e, State: Initialized, Role: FOLLOWER
I20260924 19:40:41.622838 18983 consensus_queue.cc:260] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.623045 18983 ts_tablet_manager.cc:1428] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:40:41.623091 18969 heartbeater.cc:503] Master 127.0.98.62:34455 was elected leader, sending a full tablet report...
I20260924 19:40:41.623222 18987 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Bootstrap starting.
I20260924 19:40:41.623780 18987 tablet_bootstrap.cc:654] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:41.624279 18987 log.cc:824] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:41.624866 18987 tablet_bootstrap.cc:492] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: No bootstrap required, opened a new log
I20260924 19:40:41.624915 18987 ts_tablet_manager.cc:1397] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:40:41.626187 18987 raft_consensus.cc:348] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.626315 18987 raft_consensus.cc:374] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:41.626336 18987 raft_consensus.cc:729] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7777782c9f6f49e89ddde8461aa41b64, State: Initialized, Role: FOLLOWER
I20260924 19:40:41.626412 18987 consensus_queue.cc:260] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.626616 18987 ts_tablet_manager.cc:1428] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:40:41.626631 18835 heartbeater.cc:503] Master 127.0.98.62:34455 was elected leader, sending a full tablet report...
W20260924 19:40:41.692751 18836 tablet.cc:2369] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:41.801551 18702 tablet.cc:2369] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:41.806543 18989 raft_consensus.cc:482] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:41.806669 18989 raft_consensus.cc:504] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.807060 18989 leader_election.cc:288] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1560ac3297f4279b5250ce138caebe7 (127.0.98.1:45619), 7777782c9f6f49e89ddde8461aa41b64 (127.0.98.2:43699)
I20260924 19:40:41.809655 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.807239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45619 (local address 127.0.98.3:48607)
0924 19:40:41.807274 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.807285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.807332 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.807460 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.807461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.807463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:41.807528 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.807530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.808154 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.808155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.808902 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.808904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.809422 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.809423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.809494 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.809497 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.809540 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.809576 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 19:40:41.809655 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.807282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48607 (local address 127.0.98.1:45619)
0924 19:40:41.807355 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.807357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.807367 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:40:41.807394 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.807395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.807406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.807448 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:41.807583 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.808089 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.808990 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.809354 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.809412 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.809422 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.809559 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.809593 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:40:41.809821 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.807242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43699 (local address 127.0.98.3:41589)
0924 19:40:41.807453 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.807462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:41.807521 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:41.807658 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:41.807659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:41.807661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:41.807730 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.807731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.808357 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.808358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.809082 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:41.809084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.809599 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.809600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:41.809665 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.809668 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:41.809710 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:40:41.809746 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":67,"thread_start_us":97,"threads_started":1}
I20260924 19:40:41.809872 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.807342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41589 (local address 127.0.98.2:43699)
0924 19:40:41.807383 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.807384 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.807562 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:40:41.807591 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.807592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.807602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.807641 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:41.807790 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.808289 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.809163 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.809526 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.809583 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.809592 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.809774 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.809814 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:41.809989 18655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cff128b9614e0a99f783523dd44796" candidate_uuid: "eddb0442c95d4cb78236ed426d469a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1560ac3297f4279b5250ce138caebe7" is_pre_election: true
I20260924 19:40:41.810158 18655 raft_consensus.cc:2454] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eddb0442c95d4cb78236ed426d469a4e in term 0.
I20260924 19:40:41.810256 18789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cff128b9614e0a99f783523dd44796" candidate_uuid: "eddb0442c95d4cb78236ed426d469a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7777782c9f6f49e89ddde8461aa41b64" is_pre_election: true
I20260924 19:40:41.810364 18858 leader_election.cc:302] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1560ac3297f4279b5250ce138caebe7, eddb0442c95d4cb78236ed426d469a4e; no voters:
I20260924 19:40:41.810395 18789 raft_consensus.cc:2454] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eddb0442c95d4cb78236ed426d469a4e in term 0.
I20260924 19:40:41.810482 18989 raft_consensus.cc:2781] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:41.810551 18989 raft_consensus.cc:482] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:41.810580 18989 raft_consensus.cc:3037] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:41.811256 18989 raft_consensus.cc:504] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.811367 18989 leader_election.cc:288] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [CANDIDATE]: Term 1 election: Requested vote from peers e1560ac3297f4279b5250ce138caebe7 (127.0.98.1:45619), 7777782c9f6f49e89ddde8461aa41b64 (127.0.98.2:43699)
I20260924 19:40:41.811508 18789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cff128b9614e0a99f783523dd44796" candidate_uuid: "eddb0442c95d4cb78236ed426d469a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7777782c9f6f49e89ddde8461aa41b64"
I20260924 19:40:41.811582 18789 raft_consensus.cc:3037] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:41.812371 18789 raft_consensus.cc:2454] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eddb0442c95d4cb78236ed426d469a4e in term 1.
I20260924 19:40:41.812464 18655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cff128b9614e0a99f783523dd44796" candidate_uuid: "eddb0442c95d4cb78236ed426d469a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1560ac3297f4279b5250ce138caebe7"
I20260924 19:40:41.812532 18856 leader_election.cc:302] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7777782c9f6f49e89ddde8461aa41b64, eddb0442c95d4cb78236ed426d469a4e; no voters:
I20260924 19:40:41.812542 18655 raft_consensus.cc:3037] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:41.812623 18989 raft_consensus.cc:2781] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:41.812765 18989 raft_consensus.cc:686] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 1 LEADER]: Becoming Leader. State: Replica: eddb0442c95d4cb78236ed426d469a4e, State: Running, Role: LEADER
I20260924 19:40:41.812858 18989 consensus_queue.cc:237] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } }
I20260924 19:40:41.813287 18655 raft_consensus.cc:2454] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eddb0442c95d4cb78236ed426d469a4e in term 1.
I20260924 19:40:41.813668 18510 catalog_manager.cc:5495] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to eddb0442c95d4cb78236ed426d469a4e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eddb0442c95d4cb78236ed426d469a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eddb0442c95d4cb78236ed426d469a4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } health_report { overall_health: HEALTHY } } }
W20260924 19:40:41.831509 18970 tablet.cc:2369] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:41.839823 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.837653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.98.3:38881)
0924 19:40:41.837707 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:41.837709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:41.837717 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:40:41.837754 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:41.837755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:41.837772 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:41.837829 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:41.837932 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.838390 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.839122 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:41.839298 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:41.839340 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:41.839542 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:41.839630 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:41.839702 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:41.839743 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 19:40:41.839750 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:40:41.841691 18655 raft_consensus.cc:1261] T 14cff128b9614e0a99f783523dd44796 P e1560ac3297f4279b5250ce138caebe7 [term 1 FOLLOWER]: Refusing update from remote peer eddb0442c95d4cb78236ed426d469a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:41.841691 18789 raft_consensus.cc:1261] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 1 FOLLOWER]: Refusing update from remote peer eddb0442c95d4cb78236ed426d469a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:41.842041 18993 consensus_queue.cc:1035] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1560ac3297f4279b5250ce138caebe7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:41.842183 18989 consensus_queue.cc:1035] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7777782c9f6f49e89ddde8461aa41b64" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:41.844244 18996 mvcc.cc:205] Tried to move back new op lower bound from 7332982136180682752 to 7332982136066129920. Current Snapshot: MvccSnapshot[applied={T|T < 7332982136180682752}]
I20260924 19:40:41.844627 18997 mvcc.cc:205] Tried to move back new op lower bound from 7332982136180682752 to 7332982136066129920. Current Snapshot: MvccSnapshot[applied={T|T < 7332982136180682752}]
I20260924 19:40:41.847373 18509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:41.847474 18509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:41.848273 18509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6178f1372a274c8a82b259ca7841f3e3: Sending DeleteTablet for 3 replicas of tablet 14cff128b9614e0a99f783523dd44796
I20260924 19:40:41.848569 18635 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cff128b9614e0a99f783523dd44796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58672
I20260924 19:40:41.848682 18769 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cff128b9614e0a99f783523dd44796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54286
I20260924 19:40:41.848871 18903 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cff128b9614e0a99f783523dd44796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260924 19:40:41.848891 19007 tablet_replica.cc:330] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64: stopping tablet replica
I20260924 19:40:41.848989 19007 raft_consensus.cc:2229] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:41.849107 19007 raft_consensus.cc:2258] T 14cff128b9614e0a99f783523dd44796 P 7777782c9f6f49e89ddde8461aa41b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:41.849221 19008 tablet_replica.cc:330] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: stopping tablet replica
I20260924 19:40:41.849311 19008 raft_consensus.cc:2229] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:41.849325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18572
I20260924 19:40:41.849422 19008 raft_consensus.cc:2258] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:41.851388 19008 ts_tablet_manager.cc:1905] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:41.853042 19008 ts_tablet_manager.cc:1918] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:41.853103 19008 log.cc:1196] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/14cff128b9614e0a99f783523dd44796
I20260924 19:40:41.853370 19008 ts_tablet_manager.cc:1939] T 14cff128b9614e0a99f783523dd44796 P eddb0442c95d4cb78236ed426d469a4e: Deleting consensus metadata
I20260924 19:40:41.853768 18495 catalog_manager.cc:4841] TS eddb0442c95d4cb78236ed426d469a4e (127.0.98.3:38881): tablet 14cff128b9614e0a99f783523dd44796 (table test-table [id=1e47446ff616430ca23102631a58312c]) successfully deleted
W20260924 19:40:41.854621 18496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45619 (error 108)
I20260924 19:40:41.854782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18706
I20260924 19:40:41.855898 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.855362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45619 (local address 127.0.0.1:58682)
0924 19:40:41.855815 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.855863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429}
W20260924 19:40:41.856407 18496 catalog_manager.cc:4584] TS e1560ac3297f4279b5250ce138caebe7 (127.0.98.1:45619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14cff128b9614e0a99f783523dd44796: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45619: connect: Connection refused (error 111)
I20260924 19:40:41.859256 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:41.859144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43699 (local address 127.0.0.1:54290)
0924 19:40:41.859204 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:41.859229 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:40:41.859247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18840
I20260924 19:40:41.863245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 18478
2026-09-24T19:40:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1918 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:41Z Disabled control of system clock
I20260924 19:40:41.889014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42193
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:41.967320 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:41.967545 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:41.967566 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:41.968685 19013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:41.968719 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:41.968730 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:41.968741 19013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:41.968752 19013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:41.970023 19013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:41.970206 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:41.970441 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:41.974939 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.975071 19013 server_base.cc:1034] running on GCE node
W20260924 19:40:41.975090 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:41.974943 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:41.975455 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:41.975734 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:41.976871 19013 hybrid_clock.cc:648] HybridClock initialized: now 1790278841976855 us; error 26 us; skew 500 ppm
I20260924 19:40:41.978049 19013 webserver.cc:460] Webserver started at http://127.0.98.62:37993/ using document root <none> and password file <none>
I20260924 19:40:41.978281 19013 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:41.978328 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:41.978451 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:41.979465 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "ac2bb65b0c28449e991c594a5ac2a55a"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "ebe6a684c547f497f16778d7b311a409"
server_key_iv: "d90be6c1be4208b6a7726c008878af89"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.979794 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac2bb65b0c28449e991c594a5ac2a55a"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "ebe6a684c547f497f16778d7b311a409"
server_key_iv: "d90be6c1be4208b6a7726c008878af89"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.980986 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:40:41.981678 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:41.981832 19013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:40:41.981912 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "ac2bb65b0c28449e991c594a5ac2a55a"
format_stamp: "Formatted at 2026-09-24 19:40:41 on dist-test-slave-6kf3"
server_key: "ebe6a684c547f497f16778d7b311a409"
server_key_iv: "d90be6c1be4208b6a7726c008878af89"
server_key_version: "encryptionkey@0"
I20260924 19:40:41.981966 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:41.997816 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:41.998077 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:41.998173 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:42.001868 19013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42193
I20260924 19:40:42.001907 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42193 every 8 connection(s)
I20260924 19:40:42.002202 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:42.002593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19013
I20260924 19:40:42.002673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:42.002848 392 external_mini_cluster.cc:1428] Setting key c1cc8caeef6ddebddb4d52fd993b8e23
I20260924 19:40:42.003178 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:42.006405 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: Bootstrap starting.
I20260924 19:40:42.006989 19080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:42.007274 19080 log.cc:824] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:42.008040 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: No bootstrap required, opened a new log
I20260924 19:40:42.008993 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.003345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.98.62:42193)
0924 19:40:42.003555 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.003564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.003618 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:40:42.004251 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.004260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.004340 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.004490 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:42.004689 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.006036 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.006833 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.007431 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.007541 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.008301 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:42.008305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:42.008305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:42.008493 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:42.008534 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:42.008550 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:42.008628 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.008687 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.008783 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260924 19:40:42.010048 19080 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 19:40:42.010183 19080 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:42.010205 19080 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2bb65b0c28449e991c594a5ac2a55a, State: Initialized, Role: FOLLOWER
I20260924 19:40:42.010314 19080 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 19:40:42.010382 19080 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:42.010413 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:42.010457 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:42.010936 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 19:40:42.011018 19080 leader_election.cc:302] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac2bb65b0c28449e991c594a5ac2a55a; no voters:
I20260924 19:40:42.011179 19080 leader_election.cc:288] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:42.011219 19085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:42.011338 19085 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [term 1 LEADER]: Becoming Leader. State: Replica: ac2bb65b0c28449e991c594a5ac2a55a, State: Running, Role: LEADER
I20260924 19:40:42.011418 19080 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:42.011451 19085 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 19:40:42.011786 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } } }
I20260924 19:40:42.011956 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:42.011921 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac2bb65b0c28449e991c594a5ac2a55a. Latest consensus state: current_term: 1 leader_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2bb65b0c28449e991c594a5ac2a55a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } } }
I20260924 19:40:42.012375 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:42.012297 19094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:42.012898 19094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:42.014330 19094 catalog_manager.cc:1347] Generated new cluster ID: 71cd219a13d746baab73e7d3241e3230
I20260924 19:40:42.014384 19094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:42.031985 19094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:42.032687 19094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:42.038292 19094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: Generated new TSK 0
I20260924 19:40:42.038470 19094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:42.044600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--builtin_ntp_servers=127.0.98.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:42.123078 19104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:42.123275 19104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:42.123303 19104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:42.124467 19104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:42.124505 19104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:42.124538 19104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:42.125844 19104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:42.126076 19104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:42.126346 19104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:42.132010 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.132051 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.132094 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:42.132688 19104 server_base.cc:1034] running on GCE node
I20260924 19:40:42.132894 19104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:42.133179 19104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:42.134338 19104 hybrid_clock.cc:648] HybridClock initialized: now 1790278842134324 us; error 35 us; skew 500 ppm
I20260924 19:40:42.135540 19104 webserver.cc:460] Webserver started at http://127.0.98.1:39471/ using document root <none> and password file <none>
I20260924 19:40:42.135746 19104 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:42.135793 19104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:42.135917 19104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:42.136986 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6277c7a04414212b9eeeeedc4fdb801"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "50f1773a7b22391f4eb74ad6b1f7f6f9"
server_key_iv: "fc7a7bfebb1ab7692fb9ba0b755c8eb8"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.137315 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6277c7a04414212b9eeeeedc4fdb801"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "50f1773a7b22391f4eb74ad6b1f7f6f9"
server_key_iv: "fc7a7bfebb1ab7692fb9ba0b755c8eb8"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.138523 19104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:42.139237 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.139382 19104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:42.139456 19104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "f6277c7a04414212b9eeeeedc4fdb801"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "50f1773a7b22391f4eb74ad6b1f7f6f9"
server_key_iv: "fc7a7bfebb1ab7692fb9ba0b755c8eb8"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.139525 19104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:42.148720 19104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:42.148973 19104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:42.149107 19104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:42.149304 19104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:42.149593 19104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:42.149626 19104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.149652 19104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:42.149668 19104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.155016 19104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35167
I20260924 19:40:42.155063 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35167 every 8 connection(s)
I20260924 19:40:42.155347 19104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:42.158878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19104
I20260924 19:40:42.158974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:42.159163 392 external_mini_cluster.cc:1428] Setting key 7adb5d1051081335649d60fc9bdddcd3
I20260924 19:40:42.160051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--builtin_ntp_servers=127.0.98.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:42.160327 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.155997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54643 (local address 127.0.98.62:42193)
0924 19:40:42.156055 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.156057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.156247 (+ 190us) server_negotiation.cc:407] Connection header received
0924 19:40:42.156752 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.156753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.156767 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.156835 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:42.157411 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.157902 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.158846 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.159165 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.159229 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.160067 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:42.160069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:42.160069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:42.160074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:42.160090 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:42.160095 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:42.160150 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.160232 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.160258 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:40:42.160567 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.155978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42193 (local address 127.0.98.1:54643)
0924 19:40:42.156150 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.156182 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.156661 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.156870 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.156873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.156882 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:42.157344 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.157349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.157964 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.157964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.158758 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.158761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.159260 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.159261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.159386 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.159903 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:42.159903 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:42.159966 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:42.159996 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:42.160159 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:42.160164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:42.160168 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.160234 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.160335 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260924 19:40:42.161525 19232 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42193
I20260924 19:40:42.161625 19232 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:42.161803 19232 heartbeater.cc:511] Master 127.0.98.62:42193 requested a full tablet report, sending...
I20260924 19:40:42.162367 19044 ts_manager.cc:194] Registered new tserver with Master: f6277c7a04414212b9eeeeedc4fdb801 (127.0.98.1:35167)
I20260924 19:40:42.163203 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54643
W20260924 19:40:42.237272 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:42.237453 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:42.237473 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:42.238514 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:42.238543 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:42.238565 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:42.239825 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:42.240044 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:42.240291 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:42.244853 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.244927 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.244865 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:42.245154 19235 server_base.cc:1034] running on GCE node
I20260924 19:40:42.245355 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:42.245554 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:42.246690 19235 hybrid_clock.cc:648] HybridClock initialized: now 1790278842246671 us; error 32 us; skew 500 ppm
I20260924 19:40:42.247840 19235 webserver.cc:460] Webserver started at http://127.0.98.2:33091/ using document root <none> and password file <none>
I20260924 19:40:42.248109 19235 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:42.248162 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:42.248284 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:42.249419 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "32557bc6b6db41edb55d5d59c58d741f"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "988a97265e0be9e294196bea31049589"
server_key_iv: "ac11eceadaea778e53297d875d7f866e"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.249747 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32557bc6b6db41edb55d5d59c58d741f"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "988a97265e0be9e294196bea31049589"
server_key_iv: "ac11eceadaea778e53297d875d7f866e"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.250958 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:42.251674 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.251852 19235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:42.251948 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "32557bc6b6db41edb55d5d59c58d741f"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "988a97265e0be9e294196bea31049589"
server_key_iv: "ac11eceadaea778e53297d875d7f866e"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.252012 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:42.262972 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:42.263222 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:42.263337 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:42.263531 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:42.263818 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:42.263849 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.263873 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:42.263891 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.269172 19235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36195
I20260924 19:40:42.269215 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36195 every 8 connection(s)
I20260924 19:40:42.269506 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:42.274284 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.270179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43365 (local address 127.0.98.62:42193)
0924 19:40:42.270227 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.270229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.270430 (+ 201us) server_negotiation.cc:407] Connection header received
0924 19:40:42.270898 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.270899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.270908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.270957 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:42.271567 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.272145 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.273125 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.273306 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.273352 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.274070 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:42.274072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:42.274072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:42.274076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:42.274094 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:42.274098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:42.274137 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.274203 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.274225 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:40:42.274477 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.270157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42193 (local address 127.0.98.2:43365)
0924 19:40:42.270317 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.270350 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.270820 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.271011 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.271015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.271024 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:42.271490 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.271494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.272230 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.272231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.273050 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.273053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.273367 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.273368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.273493 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.273939 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:42.273939 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:42.273971 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:42.273998 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:42.274147 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:42.274149 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:42.274152 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.274198 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.274273 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260924 19:40:42.274586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19235
I20260924 19:40:42.274681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:42.274860 392 external_mini_cluster.cc:1428] Setting key b2a0bd0c7421c3c8be3341c01b2ebfa3
I20260924 19:40:42.274874 19363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42193
I20260924 19:40:42.274964 19363 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:42.275154 19363 heartbeater.cc:511] Master 127.0.98.62:42193 requested a full tablet report, sending...
I20260924 19:40:42.275530 19044 ts_manager.cc:194] Registered new tserver with Master: 32557bc6b6db41edb55d5d59c58d741f (127.0.98.2:36195)
I20260924 19:40:42.275844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--builtin_ntp_servers=127.0.98.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:42.276006 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43365
W20260924 19:40:42.354794 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:42.354991 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:42.355011 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:42.356216 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:42.356256 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:42.356279 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:42.357558 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:42.357745 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:42.357980 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:40:42.362699 19366 server_base.cc:1034] running on GCE node
W20260924 19:40:42.362682 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.362681 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:42.362674 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:42.363041 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:42.363286 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:42.364420 19366 hybrid_clock.cc:648] HybridClock initialized: now 1790278842364401 us; error 30 us; skew 500 ppm
I20260924 19:40:42.365551 19366 webserver.cc:460] Webserver started at http://127.0.98.3:37771/ using document root <none> and password file <none>
I20260924 19:40:42.365763 19366 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:42.365810 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:42.365932 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:42.366930 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "c02b669a183d403ba77aee5a56960ad8"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "2d731869a87e3ac95c41c87a5eae0963"
server_key_iv: "28ea433761c6deffbe65e8dcbca33ffb"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.367262 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c02b669a183d403ba77aee5a56960ad8"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "2d731869a87e3ac95c41c87a5eae0963"
server_key_iv: "28ea433761c6deffbe65e8dcbca33ffb"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.368499 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:42.369155 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.369305 19366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:42.369382 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "c02b669a183d403ba77aee5a56960ad8"
format_stamp: "Formatted at 2026-09-24 19:40:42 on dist-test-slave-6kf3"
server_key: "2d731869a87e3ac95c41c87a5eae0963"
server_key_iv: "28ea433761c6deffbe65e8dcbca33ffb"
server_key_version: "encryptionkey@0"
I20260924 19:40:42.369439 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:42.385870 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:42.386204 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:42.386333 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:42.386535 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:42.386847 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:42.386885 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.386912 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:42.386929 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:42.392696 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42695
I20260924 19:40:42.392747 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42695 every 8 connection(s)
I20260924 19:40:42.393059 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:40:42.397843 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.393733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38703 (local address 127.0.98.62:42193)
0924 19:40:42.393790 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.393792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.393989 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:40:42.394453 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.394454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.394464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.394548 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:42.395195 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.395690 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.396652 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.396845 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.396900 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.397624 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:42.397626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:42.397626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:42.397630 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:42.397644 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:42.397649 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:42.397699 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.397758 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.397779 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:40:42.397997 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.393695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42193 (local address 127.0.98.3:38703)
0924 19:40:42.393869 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.393908 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.394371 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.394624 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.394628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.394636 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:42.395120 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.395125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.395761 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.395762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.396566 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.396568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.396915 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.396915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.397023 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.397481 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:42.397482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:42.397508 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:42.397545 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:42.397703 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:42.397705 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:42.397708 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.397752 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.397824 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260924 19:40:42.398429 19494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42193
I20260924 19:40:42.398527 19494 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:42.398666 19494 heartbeater.cc:511] Master 127.0.98.62:42193 requested a full tablet report, sending...
I20260924 19:40:42.399034 19044 ts_manager.cc:194] Registered new tserver with Master: c02b669a183d403ba77aee5a56960ad8 (127.0.98.3:42695)
I20260924 19:40:42.399379 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38703
I20260924 19:40:42.400496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19366
I20260924 19:40:42.400579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:42.400710 392 external_mini_cluster.cc:1428] Setting key 07593243825410e3766be25074842349
I20260924 19:40:42.401500 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:42.405324 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.402708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.98.62:42193)
0924 19:40:42.402750 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.402751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.402832 (+ 81us) server_negotiation.cc:407] Connection header received
0924 19:40:42.402874 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.402874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.402882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.402923 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:42.403061 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.403502 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.404236 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.404405 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.404451 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.405069 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:42.405070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:42.405070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:42.405074 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:42.405085 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:42.405089 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:42.405155 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.405245 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.405253 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:42.406728 19044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:42.408263 19044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:42.414499 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.1:35167)
0924 19:40:42.410859 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.410865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.410874 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:42.410914 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.410916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.410929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.410994 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:42.411174 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412280 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.413468 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414126 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414195 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414283 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.414381 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.414417 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260924 19:40:42.414502 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35167 (local address 127.0.0.1:43752)
0924 19:40:42.410568 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.410576 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.410612 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.411059 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.411060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.411062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:42.411093 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.411094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412357 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.412358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.413382 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.413385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414215 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.414283 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414294 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.414356 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.414403 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":67,"thread_start_us":58,"threads_started":1}
I20260924 19:40:42.414753 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42695 (local address 127.0.0.1:40186)
0924 19:40:42.410444 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.410480 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.410570 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.410847 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.410848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.410850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:42.410894 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.410895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412274 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.412275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.413595 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.413598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414493 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.414552 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414556 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.414644 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.414686 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":100,"thread_start_us":77,"threads_started":1}
I20260924 19:40:42.414779 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.0.98.3:42695)
0924 19:40:42.410419 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.410423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.410541 (+ 118us) server_negotiation.cc:407] Connection header received
0924 19:40:42.410656 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.410659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.410674 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.410726 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:42.410971 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412184 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.413684 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414332 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414389 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414459 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.414666 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.414704 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260924 19:40:42.415104 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.0.98.2:36195)
0924 19:40:42.410487 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.410490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.410651 (+ 161us) server_negotiation.cc:407] Connection header received
0924 19:40:42.410695 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.410697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.410711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.410772 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:42.410947 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412241 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.413691 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414155 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414213 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414296 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.414989 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.415030 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260924 19:40:42.415263 19166 tablet_service.cc:1467] Processing CreateTablet for tablet d465125bec874b54bcf4ba5d05a6cbda (DEFAULT_TABLE table=test-table [id=5c5834c23eed4eaca86b3c614b34b9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:42.415387 19428 tablet_service.cc:1467] Processing CreateTablet for tablet d465125bec874b54bcf4ba5d05a6cbda (DEFAULT_TABLE table=test-table [id=5c5834c23eed4eaca86b3c614b34b9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:42.415608 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465125bec874b54bcf4ba5d05a6cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:42.415678 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465125bec874b54bcf4ba5d05a6cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:42.416057 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.410191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36195 (local address 127.0.0.1:36700)
0924 19:40:42.410480 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.410490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.410555 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.410801 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.410804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.410805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:42.410870 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.410872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.412343 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.412344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.413595 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.413598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.414326 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.414327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.414376 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.414378 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.414418 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.414448 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260924 19:40:42.416693 19297 tablet_service.cc:1467] Processing CreateTablet for tablet d465125bec874b54bcf4ba5d05a6cbda (DEFAULT_TABLE table=test-table [id=5c5834c23eed4eaca86b3c614b34b9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:42.416991 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465125bec874b54bcf4ba5d05a6cbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:42.418118 19508 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: Bootstrap starting.
I20260924 19:40:42.418690 19508 tablet_bootstrap.cc:654] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:42.418690 19509 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Bootstrap starting.
I20260924 19:40:42.418998 19508 log.cc:824] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:42.419275 19509 tablet_bootstrap.cc:654] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:42.419509 19509 log.cc:824] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:42.419762 19508 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: No bootstrap required, opened a new log
I20260924 19:40:42.419837 19512 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Bootstrap starting.
I20260924 19:40:42.419992 19508 ts_tablet_manager.cc:1397] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:40:42.420200 19509 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: No bootstrap required, opened a new log
I20260924 19:40:42.420255 19509 ts_tablet_manager.cc:1397] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:42.420439 19512 tablet_bootstrap.cc:654] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:42.420738 19512 log.cc:824] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:42.421453 19512 tablet_bootstrap.cc:492] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: No bootstrap required, opened a new log
I20260924 19:40:42.421517 19512 ts_tablet_manager.cc:1397] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:42.422176 19509 raft_consensus.cc:348] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.422303 19509 raft_consensus.cc:374] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:42.422333 19509 raft_consensus.cc:729] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02b669a183d403ba77aee5a56960ad8, State: Initialized, Role: FOLLOWER
I20260924 19:40:42.422438 19509 consensus_queue.cc:260] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.422387 19508 raft_consensus.cc:348] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.422561 19508 raft_consensus.cc:374] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:42.422598 19508 raft_consensus.cc:729] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6277c7a04414212b9eeeeedc4fdb801, State: Initialized, Role: FOLLOWER
I20260924 19:40:42.422684 19509 ts_tablet_manager.cc:1428] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:40:42.422717 19508 consensus_queue.cc:260] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.422809 19494 heartbeater.cc:503] Master 127.0.98.62:42193 was elected leader, sending a full tablet report...
I20260924 19:40:42.422971 19508 ts_tablet_manager.cc:1428] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:40:42.423004 19232 heartbeater.cc:503] Master 127.0.98.62:42193 was elected leader, sending a full tablet report...
I20260924 19:40:42.424430 19512 raft_consensus.cc:348] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.424551 19512 raft_consensus.cc:374] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:42.424571 19512 raft_consensus.cc:729] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32557bc6b6db41edb55d5d59c58d741f, State: Initialized, Role: FOLLOWER
I20260924 19:40:42.424647 19512 consensus_queue.cc:260] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.424841 19512 ts_tablet_manager.cc:1428] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:40:42.424909 19363 heartbeater.cc:503] Master 127.0.98.62:42193 was elected leader, sending a full tablet report...
I20260924 19:40:42.515972 19514 raft_consensus.cc:482] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:42.516108 19514 raft_consensus.cc:504] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.516417 19514 leader_election.cc:288] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6277c7a04414212b9eeeeedc4fdb801 (127.0.98.1:35167), 32557bc6b6db41edb55d5d59c58d741f (127.0.98.2:36195)
I20260924 19:40:42.519140 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.516668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40247 (local address 127.0.98.2:36195)
0924 19:40:42.516710 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.516712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.516722 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:40:42.516753 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.516754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.516763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.516835 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:42.516984 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.517500 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518467 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.518833 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518889 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.518899 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.519037 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.519070 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:42.519147 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.516676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42453 (local address 127.0.98.1:35167)
0924 19:40:42.516714 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.516715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.516858 (+ 143us) server_negotiation.cc:407] Connection header received
0924 19:40:42.516911 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.516912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.516926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.516977 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:42.517139 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.517596 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518478 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.518837 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518895 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.518903 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.519034 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.519070 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:42.519162 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.516585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36195 (local address 127.0.98.3:40247)
0924 19:40:42.516643 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.516654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.516696 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.516845 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.516846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.516847 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:42.516917 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.516918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.517604 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.517605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.518385 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.518387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.518908 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.518978 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.518981 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.519026 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.519065 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":3}
I20260924 19:40:42.519479 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465125bec874b54bcf4ba5d05a6cbda" candidate_uuid: "c02b669a183d403ba77aee5a56960ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32557bc6b6db41edb55d5d59c58d741f" is_pre_election: true
I20260924 19:40:42.519610 19317 raft_consensus.cc:2454] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c02b669a183d403ba77aee5a56960ad8 in term 0.
I20260924 19:40:42.519169 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.516593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35167 (local address 127.0.98.3:42453)
0924 19:40:42.516791 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.516801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:42.516854 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:42.516990 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:42.516991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:42.516992 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:42.517078 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.517080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.517666 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.517666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.518396 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:42.518398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.518903 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.518904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:42.518971 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.518974 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:42.519024 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:40:42.519064 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260924 19:40:42.519836 19381 leader_election.cc:302] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32557bc6b6db41edb55d5d59c58d741f, c02b669a183d403ba77aee5a56960ad8; no voters:
W20260924 19:40:42.519995 19364 tablet.cc:2369] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:42.520105 19514 raft_consensus.cc:2781] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:42.520195 19514 raft_consensus.cc:482] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:42.520220 19514 raft_consensus.cc:3037] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:42.520429 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465125bec874b54bcf4ba5d05a6cbda" candidate_uuid: "c02b669a183d403ba77aee5a56960ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6277c7a04414212b9eeeeedc4fdb801" is_pre_election: true
I20260924 19:40:42.520586 19186 raft_consensus.cc:2454] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c02b669a183d403ba77aee5a56960ad8 in term 0.
I20260924 19:40:42.521039 19514 raft_consensus.cc:504] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.521154 19514 leader_election.cc:288] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [CANDIDATE]: Term 1 election: Requested vote from peers f6277c7a04414212b9eeeeedc4fdb801 (127.0.98.1:35167), 32557bc6b6db41edb55d5d59c58d741f (127.0.98.2:36195)
I20260924 19:40:42.521313 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465125bec874b54bcf4ba5d05a6cbda" candidate_uuid: "c02b669a183d403ba77aee5a56960ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6277c7a04414212b9eeeeedc4fdb801"
I20260924 19:40:42.521329 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465125bec874b54bcf4ba5d05a6cbda" candidate_uuid: "c02b669a183d403ba77aee5a56960ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32557bc6b6db41edb55d5d59c58d741f"
I20260924 19:40:42.521390 19317 raft_consensus.cc:3037] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:42.521433 19186 raft_consensus.cc:3037] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:42.522058 19317 raft_consensus.cc:2454] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c02b669a183d403ba77aee5a56960ad8 in term 1.
I20260924 19:40:42.522199 19381 leader_election.cc:302] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32557bc6b6db41edb55d5d59c58d741f, c02b669a183d403ba77aee5a56960ad8; no voters:
I20260924 19:40:42.522250 19186 raft_consensus.cc:2454] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c02b669a183d403ba77aee5a56960ad8 in term 1.
I20260924 19:40:42.522277 19514 raft_consensus.cc:2781] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:42.522399 19514 raft_consensus.cc:686] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 1 LEADER]: Becoming Leader. State: Replica: c02b669a183d403ba77aee5a56960ad8, State: Running, Role: LEADER
I20260924 19:40:42.522497 19514 consensus_queue.cc:237] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } }
I20260924 19:40:42.523233 19044 catalog_manager.cc:5495] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c02b669a183d403ba77aee5a56960ad8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c02b669a183d403ba77aee5a56960ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c02b669a183d403ba77aee5a56960ad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 } health_report { overall_health: UNKNOWN } } }
I20260924 19:40:42.530512 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.528636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.0.98.3:42695)
0924 19:40:42.528680 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:42.528683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:42.528689 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:40:42.528718 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:42.528719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:42.528734 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:42.528782 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:42.528868 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.529291 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.529964 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:42.530132 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:42.530171 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:42.530275 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:42.530352 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:42.530410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:42.530427 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 19:40:42.530433 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:40:42.532050 19317 raft_consensus.cc:1261] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 1 FOLLOWER]: Refusing update from remote peer c02b669a183d403ba77aee5a56960ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:42.532091 19186 raft_consensus.cc:1261] T d465125bec874b54bcf4ba5d05a6cbda P f6277c7a04414212b9eeeeedc4fdb801 [term 1 FOLLOWER]: Refusing update from remote peer c02b669a183d403ba77aee5a56960ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:42.532383 19518 consensus_queue.cc:1035] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6277c7a04414212b9eeeeedc4fdb801" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:42.532485 19514 consensus_queue.cc:1035] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32557bc6b6db41edb55d5d59c58d741f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:42.534744 19521 mvcc.cc:205] Tried to move back new op lower bound from 7332982139008937984 to 7332982138972635136. Current Snapshot: MvccSnapshot[applied={T|T < 7332982139008937984}]
I20260924 19:40:42.535701 19523 mvcc.cc:205] Tried to move back new op lower bound from 7332982139008937984 to 7332982138972635136. Current Snapshot: MvccSnapshot[applied={T|T < 7332982139008937984}]
I20260924 19:40:42.537096 19043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:42.537174 19043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:42.537829 19043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac2bb65b0c28449e991c594a5ac2a55a: Sending DeleteTablet for 3 replicas of tablet d465125bec874b54bcf4ba5d05a6cbda
I20260924 19:40:42.538160 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet d465125bec874b54bcf4ba5d05a6cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:42 UTC) from {username='slave'} at 127.0.0.1:40186
I20260924 19:40:42.538159 19297 tablet_service.cc:1514] Processing DeleteTablet for tablet d465125bec874b54bcf4ba5d05a6cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:42 UTC) from {username='slave'} at 127.0.0.1:36700
I20260924 19:40:42.538247 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet d465125bec874b54bcf4ba5d05a6cbda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:42 UTC) from {username='slave'} at 127.0.0.1:43752
I20260924 19:40:42.538447 19531 tablet_replica.cc:330] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: stopping tablet replica
I20260924 19:40:42.538483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19104
I20260924 19:40:42.538959 19531 raft_consensus.cc:2229] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:42.539067 19531 raft_consensus.cc:2258] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:42.539438 19531 ts_tablet_manager.cc:1905] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:42.540728 19531 ts_tablet_manager.cc:1918] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:42.540793 19531 log.cc:1196] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/d465125bec874b54bcf4ba5d05a6cbda
I20260924 19:40:42.541013 19531 ts_tablet_manager.cc:1939] T d465125bec874b54bcf4ba5d05a6cbda P 32557bc6b6db41edb55d5d59c58d741f: Deleting consensus metadata
I20260924 19:40:42.541352 19028 catalog_manager.cc:4841] TS 32557bc6b6db41edb55d5d59c58d741f (127.0.98.2:36195): tablet d465125bec874b54bcf4ba5d05a6cbda (table test-table [id=5c5834c23eed4eaca86b3c614b34b9a4]) successfully deleted
I20260924 19:40:42.544065 19532 tablet_replica.cc:330] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: stopping tablet replica
I20260924 19:40:42.544153 19532 raft_consensus.cc:2229] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:42.544266 19532 raft_consensus.cc:2258] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:42.544646 19532 ts_tablet_manager.cc:1905] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:42.546157 19532 ts_tablet_manager.cc:1918] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:42.546223 19532 log.cc:1196] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/d465125bec874b54bcf4ba5d05a6cbda
I20260924 19:40:42.546448 19532 ts_tablet_manager.cc:1939] T d465125bec874b54bcf4ba5d05a6cbda P c02b669a183d403ba77aee5a56960ad8: Deleting consensus metadata
I20260924 19:40:42.546824 19030 catalog_manager.cc:4841] TS c02b669a183d403ba77aee5a56960ad8 (127.0.98.3:42695): tablet d465125bec874b54bcf4ba5d05a6cbda (table test-table [id=5c5834c23eed4eaca86b3c614b34b9a4]) successfully deleted
W20260924 19:40:42.547116 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35167 (error 108)
I20260924 19:40:42.547361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19235
I20260924 19:40:42.547537 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:42.547422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35167 (local address 127.0.0.1:43764)
0924 19:40:42.547465 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:42.547509 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:40:42.547922 19030 catalog_manager.cc:4584] TS f6277c7a04414212b9eeeeedc4fdb801 (127.0.98.1:35167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d465125bec874b54bcf4ba5d05a6cbda: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35167: connect: Connection refused (error 111)
I20260924 19:40:42.551754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19366
I20260924 19:40:42.555744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19013
2026-09-24T19:40:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (690 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:42 dist-test-slave-6kf3 krb5kdc[19537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:42 dist-test-slave-6kf3 krb5kdc[19537](info): set up 2 sockets
Sep 24 19:40:42 dist-test-slave-6kf3 krb5kdc[19537](info): commencing operation
krb5kdc: starting...
W20260924 19:40:43.503469 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:43 dist-test-slave-6kf3 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:40:43.610771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:43.688688 19550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:43.688896 19550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:43.688926 19550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:43.690122 19550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:43.690167 19550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:43.690186 19550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:43.690202 19550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:43.690217 19550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:43.691675 19550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:43.691915 19550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:43.692149 19550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:43.696802 19558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:43.696843 19556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:43.696827 19555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:43.696884 19550 server_base.cc:1034] running on GCE node
I20260924 19:40:43.697302 19550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:43.697589 19550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:43.698720 19550 hybrid_clock.cc:648] HybridClock initialized: now 1790278843698706 us; error 30 us; skew 500 ppm
Sep 24 19:40:43 dist-test-slave-6kf3 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:43.701069 19550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:40:43.701450 19550 webserver.cc:460] Webserver started at http://127.0.98.62:40681/ using document root <none> and password file <none>
I20260924 19:40:43.701637 19550 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:43.701675 19550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:43.701763 19550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:43.702765 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "e5cd079c7d2043e9a18fc6148a4ef422"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "ddaf52786e2300dd053a420ca5e1fe3e"
server_key_iv: "eabb92831002d899f24f6ce0263579c0"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.703044 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5cd079c7d2043e9a18fc6148a4ef422"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "ddaf52786e2300dd053a420ca5e1fe3e"
server_key_iv: "eabb92831002d899f24f6ce0263579c0"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.704232 19550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:43.704883 19565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:43.705067 19550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:43.705132 19550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "e5cd079c7d2043e9a18fc6148a4ef422"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "ddaf52786e2300dd053a420ca5e1fe3e"
server_key_iv: "eabb92831002d899f24f6ce0263579c0"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.705183 19550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:43.719863 19550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:43.721812 19550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:43.721922 19550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:43.725880 19550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260924 19:40:43.725916 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260924 19:40:43.726272 19550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:43.727149 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:43.729465 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: Bootstrap starting.
I20260924 19:40:43.730022 19618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:43.730275 19618 log.cc:824] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:43.731036 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: No bootstrap required, opened a new log
I20260924 19:40:43.732693 19618 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260924 19:40:43.732836 19618 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:43.732867 19618 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cd079c7d2043e9a18fc6148a4ef422, State: Initialized, Role: FOLLOWER
I20260924 19:40:43.732999 19618 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260924 19:40:43.733060 19618 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:43.733090 19618 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:43.733135 19618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:43.733652 19618 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260924 19:40:43.733744 19618 leader_election.cc:302] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5cd079c7d2043e9a18fc6148a4ef422; no voters:
I20260924 19:40:43.733896 19618 leader_election.cc:288] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:43.733937 19621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:43.734125 19621 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [term 1 LEADER]: Becoming Leader. State: Replica: e5cd079c7d2043e9a18fc6148a4ef422, State: Running, Role: LEADER
I20260924 19:40:43.734138 19618 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:43.734272 19621 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260924 19:40:43.734748 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260924 19:40:43.734829 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:43.734930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19550
I20260924 19:40:43.734987 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5cd079c7d2043e9a18fc6148a4ef422. Latest consensus state: current_term: 1 leader_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd079c7d2043e9a18fc6148a4ef422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260924 19:40:43.735033 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:43.735036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:43.735167 19629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:43.735229 392 external_mini_cluster.cc:1428] Setting key f785785244092af72f1068268fcbd414
I20260924 19:40:43.735586 19629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:43.737164 19629 catalog_manager.cc:1347] Generated new cluster ID: 5d8c29f21e0047c5b905c1402f4ad68e
I20260924 19:40:43.737221 19629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:40:43 dist-test-slave-6kf3 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:43.744057 19629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:43.744740 19629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:43.745266 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:43.735846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.0.98.62:34329)
0924 19:40:43.736444 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:43.736453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:43.736483 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:40:43.736605 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:43.736612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:43.736666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:43.736803 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:43.737513 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.738391 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.739127 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.739545 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.739635 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:43.741792 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:43.741796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:43.741796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:43.741807 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:43.743856 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:43.744221 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:43.744224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:43.744224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:43.744322 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:43.744493 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:43.744494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:43.744494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:43.744700 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:43.744843 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:43.744920 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:40:43.745077 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":101,"threads_started":1}
I20260924 19:40:43.753072 19629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: Generated new TSK 0
I20260924 19:40:43.753237 19629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:40:43.775782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:43.854214 19644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:43.854421 19644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:43.854452 19644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:43.855593 19644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:43.855633 19644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:43.855666 19644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:43.857255 19644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:43.857484 19644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:43.857775 19644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:43.863080 19649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:43.863197 19644 server_base.cc:1034] running on GCE node
W20260924 19:40:43.863281 19650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:43.863080 19652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:43.863497 19644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:43.863734 19644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:43.864861 19644 hybrid_clock.cc:648] HybridClock initialized: now 1790278843864848 us; error 27 us; skew 500 ppm
Sep 24 19:40:43 dist-test-slave-6kf3 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:43.867163 19644 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:40:43.867559 19644 webserver.cc:460] Webserver started at http://127.0.98.1:35101/ using document root <none> and password file <none>
I20260924 19:40:43.867748 19644 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:43.867794 19644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:43.867913 19644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:43.868925 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "f142732a277648628229e4724af3e497"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "911395bbbce71ee91bb648cc59845c7c"
server_key_iv: "ddbf809737fb14856be8b26b0c183f27"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.869246 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f142732a277648628229e4724af3e497"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "911395bbbce71ee91bb648cc59845c7c"
server_key_iv: "ddbf809737fb14856be8b26b0c183f27"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.870390 19644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:40:43.871053 19659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:43.871217 19644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:40:43.871292 19644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "f142732a277648628229e4724af3e497"
format_stamp: "Formatted at 2026-09-24 19:40:43 on dist-test-slave-6kf3"
server_key: "911395bbbce71ee91bb648cc59845c7c"
server_key_iv: "ddbf809737fb14856be8b26b0c183f27"
server_key_version: "encryptionkey@0"
I20260924 19:40:43.871356 19644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:43.886745 19644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:43.888540 19644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:43.888664 19644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:43.888852 19644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:43.889134 19644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:43.889165 19644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:43.889194 19644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:43.889217 19644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:43.894454 19644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43107
I20260924 19:40:43.894519 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43107 every 8 connection(s)
I20260924 19:40:43.894783 19644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:40:43 dist-test-slave-6kf3 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:43.900682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19644
I20260924 19:40:43.900791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:43.900967 392 external_mini_cluster.cc:1428] Setting key bb39bf9196cd34c3319c62e673ae7656
I20260924 19:40:43.903417 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:43.895433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44201 (local address 127.0.98.62:34329)
0924 19:40:43.895486 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:43.895488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:43.895669 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:40:43.896641 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:43.896642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:43.896655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:43.896705 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:43.897312 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.897815 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.898786 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.898995 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.899053 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:43.900941 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:43.900942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:43.900943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:43.900950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:43.902196 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:43.902559 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:43.902560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:43.902561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:43.902613 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:43.902821 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:43.902822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:43.902822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:43.902936 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:43.902989 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:43.903243 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:40:43.903331 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:40:43.903564 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:43.895404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.1:44201)
0924 19:40:43.895554 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:43.895590 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:43.896535 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:43.896773 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:43.896777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:43.897025 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:43.897219 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:43.897224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.897892 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.897892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:43.898702 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:43.898705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:43.899073 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:43.899074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:43.899185 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:43.899587 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:43.899594 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:43.900860 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:43.902268 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:43.902274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:43.902276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:43.902476 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:43.902665 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:43.902666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:43.902666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:43.902765 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:43.902987 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:43.902993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:43.903128 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:40:43.903213 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:40:43.903358 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260924 19:40:43.904008 19773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260924 19:40:43.904098 19773 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:43.904264 19773 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260924 19:40:43.904745 19582 ts_manager.cc:194] Registered new tserver with Master: f142732a277648628229e4724af3e497 (127.0.98.1:43107)
I20260924 19:40:43.905339 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44201
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:40:43.922461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:44.003271 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.003459 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.003481 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.004824 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.004882 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.004912 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:44.006465 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.006690 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.006894 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:44.011642 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.011754 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.011659 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.011869 19778 server_base.cc:1034] running on GCE node
I20260924 19:40:44.012097 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.012308 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.013447 19778 hybrid_clock.cc:648] HybridClock initialized: now 1790278844013429 us; error 31 us; skew 500 ppm
Sep 24 19:40:44 dist-test-slave-6kf3 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:44.015722 19778 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:40:44.016161 19778 webserver.cc:460] Webserver started at http://127.0.98.2:34449/ using document root <none> and password file <none>
I20260924 19:40:44.016371 19778 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.016417 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.016537 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.017520 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "4ac3b6f7e30b49d4a1a18a3631e15805"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "099eb56afc2b8ac7a06c2c3ab0cba863"
server_key_iv: "78fc8f6f7f6ef7e699885ae310e482ad"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.017834 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4ac3b6f7e30b49d4a1a18a3631e15805"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "099eb56afc2b8ac7a06c2c3ab0cba863"
server_key_iv: "78fc8f6f7f6ef7e699885ae310e482ad"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.019073 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:44.019711 19793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.019853 19778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:44.019922 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "4ac3b6f7e30b49d4a1a18a3631e15805"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "099eb56afc2b8ac7a06c2c3ab0cba863"
server_key_iv: "78fc8f6f7f6ef7e699885ae310e482ad"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.020004 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.027129 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.028714 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.028831 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.029021 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:44.029279 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:44.029310 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.029336 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:44.029361 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.034574 19778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33117
I20260924 19:40:44.034636 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33117 every 8 connection(s)
I20260924 19:40:44.034926 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:44.036624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19778
I20260924 19:40:44.036720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:44.036887 392 external_mini_cluster.cc:1428] Setting key 23b49f40d601a0ed8a4606109ae18249
Sep 24 19:40:44 dist-test-slave-6kf3 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:44.044718 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.035566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47523 (local address 127.0.98.62:34329)
0924 19:40:44.035629 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.035631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.035815 (+ 184us) server_negotiation.cc:407] Connection header received
0924 19:40:44.036742 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.036743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.036755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.036827 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.037564 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.038250 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.039521 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.039711 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.039756 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.042143 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.042145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.042145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:44.042152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.043387 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.043805 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.043806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.043806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.043849 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.044147 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.044147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.044147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.044242 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.044288 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.044598 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.044655 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:40:44.044940 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.035560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.2:47523)
0924 19:40:44.035702 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.035741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.036640 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.036888 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.036891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.037217 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:44.037467 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.037473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.038341 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.038342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.039433 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.039437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.039780 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.039781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.039916 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.040404 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:44.040414 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:44.042058 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:44.043466 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:44.043472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:44.043475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:44.043741 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:44.043895 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:44.043895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:44.043896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:44.044089 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:44.044300 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:44.044305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:44.044464 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.044571 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.044725 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260924 19:40:44.045339 19907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260924 19:40:44.045418 19907 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:44.045573 19907 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260924 19:40:44.045877 19582 ts_manager.cc:194] Registered new tserver with Master: 4ac3b6f7e30b49d4a1a18a3631e15805 (127.0.98.2:33117)
I20260924 19:40:44.046247 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:40:44.059010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:44.136785 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.136976 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.136999 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.138154 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.138191 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.138216 19912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:44.139645 19912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.139825 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.140025 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:40:44.144727 19912 server_base.cc:1034] running on GCE node
W20260924 19:40:44.144704 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.144690 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.144694 19920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.145162 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.145421 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.146559 19912 hybrid_clock.cc:648] HybridClock initialized: now 1790278844146547 us; error 27 us; skew 500 ppm
Sep 24 19:40:44 dist-test-slave-6kf3 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:44.149124 19912 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:40:44.149562 19912 webserver.cc:460] Webserver started at http://127.0.98.3:45841/ using document root <none> and password file <none>
I20260924 19:40:44.149794 19912 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.149847 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.149968 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.151134 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d0977501356487db22d93eb48ab1a6c"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "421d73808544e5062854522e9a12c355"
server_key_iv: "c1a4334629234a280cdfa54af742888c"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.151422 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d0977501356487db22d93eb48ab1a6c"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "421d73808544e5062854522e9a12c355"
server_key_iv: "c1a4334629234a280cdfa54af742888c"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.152666 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:44.153321 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.153460 19912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:44.153530 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "0d0977501356487db22d93eb48ab1a6c"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "421d73808544e5062854522e9a12c355"
server_key_iv: "c1a4334629234a280cdfa54af742888c"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.153605 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.165082 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.167078 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.167198 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.167411 19912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:44.167689 19912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:44.167721 19912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.167742 19912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:44.167752 19912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.172927 19912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41171
I20260924 19:40:44.172995 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41171 every 8 connection(s)
I20260924 19:40:44.173259 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:40:44 dist-test-slave-6kf3 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:40:44.181579 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.173885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35309 (local address 127.0.98.62:34329)
0924 19:40:44.173942 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.173944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.174160 (+ 216us) server_negotiation.cc:407] Connection header received
0924 19:40:44.175088 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.175089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.175101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.175150 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.175740 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.176275 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.177248 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.177432 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.177486 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.179409 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.179411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.179411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:44.179417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.180559 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.180895 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.180896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.180896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.180939 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.181136 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.181136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.181136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.181225 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.181268 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.181466 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.181518 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:40:44.181744 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.173861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.3:35309)
0924 19:40:44.174034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.174078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.174964 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.175224 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.175228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.175472 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:44.175656 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.175660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.176359 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.176360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.177170 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.177172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.177497 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.177497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.177618 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.178033 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:40:44.178040 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:44.179336 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:44.180638 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:44.180644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:44.180645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:44.180842 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:44.180980 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:40:44.180980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:40:44.180980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:40:44.181088 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:40:44.181266 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:44.181269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:44.181378 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.181449 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.181564 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260924 19:40:44.182164 20041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260924 19:40:44.182245 20041 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:44.182400 20041 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260924 19:40:44.182746 19582 ts_manager.cc:194] Registered new tserver with Master: 0d0977501356487db22d93eb48ab1a6c (127.0.98.3:41171)
I20260924 19:40:44.182950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 19912
I20260924 19:40:44.183055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:44.183218 392 external_mini_cluster.cc:1428] Setting key 683759aaaf6ecf2c027e7804b038e97f
I20260924 19:40:44.183204 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35309
I20260924 19:40:44.184055 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:44.190532 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.185298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.0.98.62:34329)
0924 19:40:44.185355 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.185357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.185429 (+ 72us) server_negotiation.cc:407] Connection header received
0924 19:40:44.185499 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.185500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.185508 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.185558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.186062 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.186535 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.187234 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.187427 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.187477 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.188546 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.188547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.188547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:44.188554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.189573 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.189865 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.189866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.189866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.189915 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:44.190064 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:44.190064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:44.190065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:44.190198 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.190282 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.190383 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.190450 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:40:44.191921 19582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:44.193423 19582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:44.199071 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41171 (local address 127.0.0.1:58350)
0924 19:40:44.195616 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.195644 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.195714 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.195882 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.195885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.195886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.195974 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.195976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.196928 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.196929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.197930 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.197933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.198826 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.198827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.198881 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.198889 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.198941 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.198989 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":49,"threads_started":1}
I20260924 19:40:44.199082 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.0.98.3:41171)
0924 19:40:44.195568 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.195572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.195710 (+ 138us) server_negotiation.cc:407] Connection header received
0924 19:40:44.195784 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.195786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.195798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.195871 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.196119 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.196837 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.198031 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.198716 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.198747 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.198784 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.198970 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.199006 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260924 19:40:44.199707 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33117 (local address 127.0.0.1:36088)
0924 19:40:44.195810 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.195818 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.195858 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.196047 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.196048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.196051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.196101 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.196104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.197418 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.197419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.198530 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.198533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.199450 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.199451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.199511 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.199536 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.199591 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.199628 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":29,"thread_start_us":96,"threads_started":1}
I20260924 19:40:44.199726 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.98.2:33117)
0924 19:40:44.195592 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.195597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.195869 (+ 272us) server_negotiation.cc:407] Connection header received
0924 19:40:44.195913 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.195915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.195950 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.196170 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.196250 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.197317 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.198610 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.199266 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.199329 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.199380 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.199620 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.199655 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260924 19:40:44.199842 19975 tablet_service.cc:1467] Processing CreateTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab (DEFAULT_TABLE table=test-table [id=29c6dd6e1a694e4e880952513c039191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.200218 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa640d48a1e044dd8ba54a79af32d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.200484 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43107 (local address 127.0.0.1:50130)
0924 19:40:44.195922 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.195952 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.195997 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.196308 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.196309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.196310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.196340 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.196341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.197489 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.197490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.198319 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.198321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.199493 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.199494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.199537 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.199541 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.199573 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.199610 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":55,"threads_started":1}
I20260924 19:40:44.200459 19841 tablet_service.cc:1467] Processing CreateTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab (DEFAULT_TABLE table=test-table [id=29c6dd6e1a694e4e880952513c039191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.200595 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.195562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.0.98.1:43107)
0924 19:40:44.195758 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.195762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.195997 (+ 235us) server_negotiation.cc:407] Connection header received
0924 19:40:44.196060 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.196062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.196077 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.196229 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.196407 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.197419 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.198383 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.199009 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.199070 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.199682 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.199704 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.200095 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260924 19:40:44.200801 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa640d48a1e044dd8ba54a79af32d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.201169 19707 tablet_service.cc:1467] Processing CreateTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab (DEFAULT_TABLE table=test-table [id=29c6dd6e1a694e4e880952513c039191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.201563 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa640d48a1e044dd8ba54a79af32d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.203316 20056 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Bootstrap starting.
I20260924 19:40:44.203397 20055 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Bootstrap starting.
I20260924 19:40:44.203995 20056 tablet_bootstrap.cc:654] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.204005 20055 tablet_bootstrap.cc:654] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.204365 20057 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: Bootstrap starting.
I20260924 19:40:44.204556 20055 log.cc:824] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.204556 20056 log.cc:824] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.204921 20057 tablet_bootstrap.cc:654] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.205267 20055 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: No bootstrap required, opened a new log
I20260924 19:40:44.205334 20055 ts_tablet_manager.cc:1397] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:40:44.205412 20057 log.cc:824] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.205489 20056 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: No bootstrap required, opened a new log
I20260924 19:40:44.205549 20056 ts_tablet_manager.cc:1397] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:40:44.205983 20057 tablet_bootstrap.cc:492] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: No bootstrap required, opened a new log
I20260924 19:40:44.206045 20057 ts_tablet_manager.cc:1397] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:44.207357 20056 raft_consensus.cc:348] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207387 20055 raft_consensus.cc:348] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207531 20056 raft_consensus.cc:374] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.207531 20055 raft_consensus.cc:374] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.207566 20055 raft_consensus.cc:729] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac3b6f7e30b49d4a1a18a3631e15805, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.207566 20056 raft_consensus.cc:729] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0977501356487db22d93eb48ab1a6c, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.207528 20057 raft_consensus.cc:348] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207690 20057 raft_consensus.cc:374] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.207726 20057 raft_consensus.cc:729] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f142732a277648628229e4724af3e497, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.207700 20055 consensus_queue.cc:260] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207711 20056 consensus_queue.cc:260] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207818 20057 consensus_queue.cc:260] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.207979 20056 ts_tablet_manager.cc:1428] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:40:44.207979 20055 ts_tablet_manager.cc:1428] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:40:44.208081 20057 ts_tablet_manager.cc:1428] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:40:44.208101 19907 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260924 19:40:44.208098 20041 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260924 19:40:44.208473 19773 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
W20260924 19:40:44.285379 19908 tablet.cc:2369] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:44.301419 20063 raft_consensus.cc:482] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:44.301558 20063 raft_consensus.cc:504] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.301908 20063 leader_election.cc:288] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0977501356487db22d93eb48ab1a6c (127.0.98.3:41171), 4ac3b6f7e30b49d4a1a18a3631e15805 (127.0.98.2:33117)
I20260924 19:40:44.305783 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.302170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33041 (local address 127.0.98.2:33117)
0924 19:40:44.302227 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.302229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.302242 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:40:44.302272 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.302273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.302281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.302330 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.302487 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.303299 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.304835 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.305437 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.305496 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.305506 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.305670 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.305711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:40:44.305783 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.302156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52505 (local address 127.0.98.3:41171)
0924 19:40:44.302211 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.302213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.302398 (+ 185us) server_negotiation.cc:407] Connection header received
0924 19:40:44.302422 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.302423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.302430 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.302485 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.302608 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.303291 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.304834 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.305408 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.305496 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.305506 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.305670 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.305711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:40:44.305794 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.302085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33117 (local address 127.0.98.1:33041)
0924 19:40:44.302162 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.302173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.302220 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.302352 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.302353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.302354 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.302429 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.302431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.303383 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.303384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.304734 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.304737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.305510 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.305511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.305604 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.305608 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.305660 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.305702 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":81}
I20260924 19:40:44.305799 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.302103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41171 (local address 127.0.98.1:52505)
0924 19:40:44.302295 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.302302 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.302352 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.302496 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.302497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.302498 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.302556 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.302557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.303382 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.303384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.304734 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.304737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.305508 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.305509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.305604 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.305608 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.305660 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.305702 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260924 19:40:44.306116 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa640d48a1e044dd8ba54a79af32d9ab" candidate_uuid: "f142732a277648628229e4724af3e497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" is_pre_election: true
I20260924 19:40:44.306282 19861 raft_consensus.cc:2454] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f142732a277648628229e4724af3e497 in term 0.
I20260924 19:40:44.306376 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa640d48a1e044dd8ba54a79af32d9ab" candidate_uuid: "f142732a277648628229e4724af3e497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0977501356487db22d93eb48ab1a6c" is_pre_election: true
I20260924 19:40:44.306493 19661 leader_election.cc:302] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac3b6f7e30b49d4a1a18a3631e15805, f142732a277648628229e4724af3e497; no voters:
I20260924 19:40:44.306524 19995 raft_consensus.cc:2454] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f142732a277648628229e4724af3e497 in term 0.
I20260924 19:40:44.306619 20063 raft_consensus.cc:2781] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:44.306694 20063 raft_consensus.cc:482] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:44.306720 20063 raft_consensus.cc:3037] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:44.307411 20063 raft_consensus.cc:504] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.307538 20063 leader_election.cc:288] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [CANDIDATE]: Term 1 election: Requested vote from peers 0d0977501356487db22d93eb48ab1a6c (127.0.98.3:41171), 4ac3b6f7e30b49d4a1a18a3631e15805 (127.0.98.2:33117)
I20260924 19:40:44.307684 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa640d48a1e044dd8ba54a79af32d9ab" candidate_uuid: "f142732a277648628229e4724af3e497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0977501356487db22d93eb48ab1a6c"
I20260924 19:40:44.307708 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa640d48a1e044dd8ba54a79af32d9ab" candidate_uuid: "f142732a277648628229e4724af3e497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805"
I20260924 19:40:44.307752 19995 raft_consensus.cc:3037] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:44.307770 19861 raft_consensus.cc:3037] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:44.308497 19861 raft_consensus.cc:2454] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f142732a277648628229e4724af3e497 in term 1.
I20260924 19:40:44.308648 19661 leader_election.cc:302] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac3b6f7e30b49d4a1a18a3631e15805, f142732a277648628229e4724af3e497; no voters:
I20260924 19:40:44.308676 19995 raft_consensus.cc:2454] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f142732a277648628229e4724af3e497 in term 1.
I20260924 19:40:44.308729 20063 raft_consensus.cc:2781] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:44.308876 20063 raft_consensus.cc:686] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 1 LEADER]: Becoming Leader. State: Replica: f142732a277648628229e4724af3e497, State: Running, Role: LEADER
I20260924 19:40:44.308970 20063 consensus_queue.cc:237] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } }
I20260924 19:40:44.309835 19582 catalog_manager.cc:5495] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 reported cstate change: term changed from 0 to 1, leader changed from <none> to f142732a277648628229e4724af3e497 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f142732a277648628229e4724af3e497" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f142732a277648628229e4724af3e497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43107 } health_report { overall_health: HEALTHY } } }
I20260924 19:40:44.343024 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.340572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.1:43107)
0924 19:40:44.340627 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.340629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.340637 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:40:44.340698 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.340699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.340721 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.340788 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:44.340914 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.341508 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.342315 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.342504 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.342557 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.342708 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:44.342810 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:44.342882 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.342899 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.342920 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:40:44.344877 19861 raft_consensus.cc:1261] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 1 FOLLOWER]: Refusing update from remote peer f142732a277648628229e4724af3e497: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:44.344877 19995 raft_consensus.cc:1261] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 1 FOLLOWER]: Refusing update from remote peer f142732a277648628229e4724af3e497: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:44.345253 20065 consensus_queue.cc:1035] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac3b6f7e30b49d4a1a18a3631e15805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:44.345382 20063 consensus_queue.cc:1035] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0977501356487db22d93eb48ab1a6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:44.348163 20069 mvcc.cc:205] Tried to move back new op lower bound from 7332982146433892352 to 7332982146290212864. Current Snapshot: MvccSnapshot[applied={T|T < 7332982146433892352 or (T in {7332982146433892352})}]
I20260924 19:40:44.348554 20070 mvcc.cc:205] Tried to move back new op lower bound from 7332982146433892352 to 7332982146290212864. Current Snapshot: MvccSnapshot[applied={T|T < 7332982146433892352}]
I20260924 19:40:44.350183 19582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:44.350282 19582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:44.351070 19582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cd079c7d2043e9a18fc6148a4ef422: Sending DeleteTablet for 3 replicas of tablet aa640d48a1e044dd8ba54a79af32d9ab
I20260924 19:40:44.351372 19707 tablet_service.cc:1514] Processing DeleteTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50130
I20260924 19:40:44.351372 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58350
I20260924 19:40:44.351410 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet aa640d48a1e044dd8ba54a79af32d9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36088
I20260924 19:40:44.351577 20078 tablet_replica.cc:330] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497: stopping tablet replica
I20260924 19:40:44.351662 20078 raft_consensus.cc:2229] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:44.351775 20078 raft_consensus.cc:2258] T aa640d48a1e044dd8ba54a79af32d9ab P f142732a277648628229e4724af3e497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:44.351873 20079 tablet_replica.cc:330] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: stopping tablet replica
I20260924 19:40:44.351979 20079 raft_consensus.cc:2229] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:44.352052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19644
I20260924 19:40:44.352072 20079 raft_consensus.cc:2258] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:44.352978 20079 ts_tablet_manager.cc:1905] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:44.353467 20080 tablet_replica.cc:330] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: stopping tablet replica
I20260924 19:40:44.353576 20080 raft_consensus.cc:2229] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:44.353667 20080 raft_consensus.cc:2258] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:44.354079 20080 ts_tablet_manager.cc:1905] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:44.354640 20079 ts_tablet_manager.cc:1918] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:44.354705 20079 log.cc:1196] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/aa640d48a1e044dd8ba54a79af32d9ab
I20260924 19:40:44.354931 20079 ts_tablet_manager.cc:1939] T aa640d48a1e044dd8ba54a79af32d9ab P 0d0977501356487db22d93eb48ab1a6c: Deleting consensus metadata
I20260924 19:40:44.355291 19566 catalog_manager.cc:4841] TS 0d0977501356487db22d93eb48ab1a6c (127.0.98.3:41171): tablet aa640d48a1e044dd8ba54a79af32d9ab (table test-table [id=29c6dd6e1a694e4e880952513c039191]) successfully deleted
I20260924 19:40:44.355505 20080 ts_tablet_manager.cc:1918] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:44.355571 20080 log.cc:1196] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/aa640d48a1e044dd8ba54a79af32d9ab
I20260924 19:40:44.355798 20080 ts_tablet_manager.cc:1939] T aa640d48a1e044dd8ba54a79af32d9ab P 4ac3b6f7e30b49d4a1a18a3631e15805: Deleting consensus metadata
I20260924 19:40:44.356171 19567 catalog_manager.cc:4841] TS 4ac3b6f7e30b49d4a1a18a3631e15805 (127.0.98.2:33117): tablet aa640d48a1e044dd8ba54a79af32d9ab (table test-table [id=29c6dd6e1a694e4e880952513c039191]) successfully deleted
W20260924 19:40:44.357376 19567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43107 (error 108)
I20260924 19:40:44.357585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19778
I20260924 19:40:44.357792 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.357695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43107 (local address 127.0.0.1:50150)
0924 19:40:44.357735 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.357765 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:40:44.357879 19567 catalog_manager.cc:4584] TS f142732a277648628229e4724af3e497 (127.0.98.1:43107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa640d48a1e044dd8ba54a79af32d9ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43107: connect: Connection refused (error 111)
I20260924 19:40:44.362257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19912
I20260924 19:40:44.366050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 19550
2026-09-24T19:40:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1811 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:44Z Disabled control of system clock
I20260924 19:40:44.390728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39079
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:44.467926 20085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.468155 20085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.468175 20085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.469269 20085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:44.469312 20085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.469326 20085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.469336 20085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:44.469349 20085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:44.470687 20085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.470885 20085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.471117 20085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:44.475579 20091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.475749 20085 server_base.cc:1034] running on GCE node
W20260924 19:40:44.475584 20093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.475559 20090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.476117 20085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.476382 20085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.477535 20085 hybrid_clock.cc:648] HybridClock initialized: now 1790278844477517 us; error 30 us; skew 500 ppm
I20260924 19:40:44.478753 20085 webserver.cc:460] Webserver started at http://127.0.98.62:41261/ using document root <none> and password file <none>
I20260924 19:40:44.478967 20085 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.479017 20085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.479131 20085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.480212 20085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "25b5322b70e669c12fbae7ed7ec39150"
server_key_iv: "2b0243ab83769cff93aaed3781ef1439"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.480535 20085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "25b5322b70e669c12fbae7ed7ec39150"
server_key_iv: "2b0243ab83769cff93aaed3781ef1439"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.481762 20085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:44.482391 20099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.482547 20085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:44.482625 20085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "25b5322b70e669c12fbae7ed7ec39150"
server_key_iv: "2b0243ab83769cff93aaed3781ef1439"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.482682 20085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.495553 20085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.495852 20085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.496035 20085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.499508 20085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39079
I20260924 19:40:44.499544 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39079 every 8 connection(s)
I20260924 19:40:44.499852 20085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:44.500790 20152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.503002 20152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: Bootstrap starting.
I20260924 19:40:44.503543 20152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.503801 20152 log.cc:824] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.504459 20152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: No bootstrap required, opened a new log
I20260924 19:40:44.504535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20085
I20260924 19:40:44.504633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:44.504791 392 external_mini_cluster.cc:1428] Setting key 0f9f18015acc43eb0590cdc754e9bb7a
I20260924 19:40:44.506271 20152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } }
I20260924 19:40:44.506410 20152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.506441 20152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b2cd5b4d434d51bbee1bafd9fc6ec6, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.506592 20152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } }
I20260924 19:40:44.506660 20152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:44.506690 20152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:44.506736 20152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:44.507259 20152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } }
I20260924 19:40:44.507352 20152 leader_election.cc:302] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b2cd5b4d434d51bbee1bafd9fc6ec6; no voters:
I20260924 19:40:44.507509 20152 leader_election.cc:288] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:44.507545 20157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:44.507688 20157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 43b2cd5b4d434d51bbee1bafd9fc6ec6, State: Running, Role: LEADER
I20260924 19:40:44.507781 20152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:44.507790 20157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } }
I20260924 19:40:44.508193 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b2cd5b4d434d51bbee1bafd9fc6ec6. Latest consensus state: current_term: 1 leader_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } } }
I20260924 19:40:44.508265 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:44.508258 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b2cd5b4d434d51bbee1bafd9fc6ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39079 } } }
I20260924 19:40:44.508344 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:44.508564 20164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:44.509064 20164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:44.510807 20164 catalog_manager.cc:1347] Generated new cluster ID: 2a6a154f362b4704b5b6977ef3591a65
I20260924 19:40:44.510850 20164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:44.511644 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.505294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.0.98.62:39079)
0924 19:40:44.505489 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.505496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.505523 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:40:44.506244 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.506251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.506310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.506430 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.506693 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.508290 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.509466 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.510087 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.510203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.510948 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.510952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.510952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:44.511152 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.511191 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:44.511207 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.511273 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.511365 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.511444 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260924 19:40:44.518337 20164 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:44.519048 20164 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:44.524077 20164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: Generated new TSK 0
I20260924 19:40:44.524250 20164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:44.528766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--builtin_ntp_servers=127.0.98.20:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:40:44.606359 20176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.606554 20176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.606575 20176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.607712 20176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.607750 20176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.607774 20176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:40:44.609177 20176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.609395 20176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.609691 20176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:44.614667 20184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.614696 20182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.614696 20181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.614929 20176 server_base.cc:1034] running on GCE node
I20260924 19:40:44.615165 20176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.615391 20176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.616535 20176 hybrid_clock.cc:648] HybridClock initialized: now 1790278844616519 us; error 39 us; skew 500 ppm
I20260924 19:40:44.617760 20176 webserver.cc:460] Webserver started at http://127.0.98.1:35919/ using document root <none> and password file <none>
I20260924 19:40:44.617985 20176 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.618029 20176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.618149 20176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.619161 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data/instance:
uuid: "8135771d50b54522aa0c6b7eada62487"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "655435ecec5b0cb76c5fa14b12b861b0"
server_key_iv: "ecb19364c6ae2e659c6155b4a8309b62"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.619473 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8135771d50b54522aa0c6b7eada62487"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "655435ecec5b0cb76c5fa14b12b861b0"
server_key_iv: "ecb19364c6ae2e659c6155b4a8309b62"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.620688 20176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:40:44.621340 20190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.621521 20176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:40:44.621615 20176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal
uuid: "8135771d50b54522aa0c6b7eada62487"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "655435ecec5b0cb76c5fa14b12b861b0"
server_key_iv: "ecb19364c6ae2e659c6155b4a8309b62"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.621683 20176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.639729 20176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.640043 20176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.640158 20176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.640337 20176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:44.640596 20176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:44.640625 20176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.640652 20176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:44.640667 20176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.645941 20176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44965
I20260924 19:40:44.646021 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44965 every 8 connection(s)
I20260924 19:40:44.646337 20176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:40:44.652621 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.647052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33255 (local address 127.0.98.62:39079)
0924 19:40:44.647116 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.647118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.647424 (+ 306us) server_negotiation.cc:407] Connection header received
0924 19:40:44.648087 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.648089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.648106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.648172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.648962 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.649512 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.650909 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.651198 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.651265 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.652214 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.652216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.652216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:44.652221 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.652238 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:44.652243 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.652310 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.652480 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.652537 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:40:44.652881 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.647062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39079 (local address 127.0.98.1:33255)
0924 19:40:44.647278 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.647322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.647951 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.648244 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.648248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.648259 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:44.648880 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.648885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.649621 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.649623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.650807 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.650810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.651297 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.651298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.651445 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.652060 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:44.652060 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:44.652098 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:44.652125 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:44.652342 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:44.652348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:44.652351 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.652475 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.652588 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260924 19:40:44.653376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20176
I20260924 19:40:44.653465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:40:44.653656 392 external_mini_cluster.cc:1428] Setting key 4f7e1fc6c671269d46758b6138924b9a
I20260924 19:40:44.653949 20304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39079
I20260924 19:40:44.654039 20304 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:44.654241 20304 heartbeater.cc:511] Master 127.0.98.62:39079 requested a full tablet report, sending...
I20260924 19:40:44.654627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--builtin_ntp_servers=127.0.98.20:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:44.654780 20116 ts_manager.cc:194] Registered new tserver with Master: 8135771d50b54522aa0c6b7eada62487 (127.0.98.1:44965)
I20260924 19:40:44.655381 20116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33255
W20260924 19:40:44.733768 20307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.733966 20307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.733994 20307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.735138 20307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.735177 20307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.735201 20307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:40:44.736529 20307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.736750 20307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.737032 20307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:44.741788 20315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.741829 20312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.741830 20313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.742018 20307 server_base.cc:1034] running on GCE node
I20260924 19:40:44.742210 20307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.742429 20307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.743570 20307 hybrid_clock.cc:648] HybridClock initialized: now 1790278844743544 us; error 36 us; skew 500 ppm
I20260924 19:40:44.744757 20307 webserver.cc:460] Webserver started at http://127.0.98.2:42523/ using document root <none> and password file <none>
I20260924 19:40:44.744972 20307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.745051 20307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.745180 20307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.746205 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data/instance:
uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e6270a4424eb1b90b075befe7a977ece"
server_key_iv: "9a9b4ed4e1e1a9c0a5971314d5196cf8"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.746518 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e6270a4424eb1b90b075befe7a977ece"
server_key_iv: "9a9b4ed4e1e1a9c0a5971314d5196cf8"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.747712 20307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:44.748567 20321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.748709 20307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:40:44.748786 20307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal
uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e6270a4424eb1b90b075befe7a977ece"
server_key_iv: "9a9b4ed4e1e1a9c0a5971314d5196cf8"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.748852 20307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.757083 20307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.757340 20307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.757454 20307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.757658 20307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:44.757936 20307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:44.757967 20307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.757992 20307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:44.758049 20307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.763360 20307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39197
I20260924 19:40:44.763408 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39197 every 8 connection(s)
I20260924 19:40:44.763710 20307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:40:44.768571 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.764391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60441 (local address 127.0.98.62:39079)
0924 19:40:44.764435 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.764437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.764666 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:40:44.765156 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.765157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.765169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.765221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.765876 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.766371 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.767301 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.767485 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.767538 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.768322 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.768324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.768324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:44.768329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.768343 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:44.768347 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.768390 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.768467 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.768494 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:40:44.768775 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.764366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39079 (local address 127.0.98.2:60441)
0924 19:40:44.764539 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.764570 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.765063 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.765296 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.765300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.765308 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:44.765796 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.765800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.766446 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.766447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.767221 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.767224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.767548 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.767549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.767675 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.768191 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:44.768191 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:44.768221 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:44.768242 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:44.768395 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:44.768400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:44.768403 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.768464 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.768563 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260924 19:40:44.769114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20307
I20260924 19:40:44.769214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:40:44.769243 20435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39079
I20260924 19:40:44.769330 20435 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:44.769415 392 external_mini_cluster.cc:1428] Setting key cc0d206e0ec131ba9a5f94d450bd54e4
I20260924 19:40:44.769517 20435 heartbeater.cc:511] Master 127.0.98.62:39079 requested a full tablet report, sending...
I20260924 19:40:44.769881 20115 ts_manager.cc:194] Registered new tserver with Master: 4dd97e4c3ca64de4864f3ef87ddbc9d2 (127.0.98.2:39197)
I20260924 19:40:44.770229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--builtin_ntp_servers=127.0.98.20:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:40:44.770256 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60441
W20260924 19:40:44.848160 20438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:44.848351 20438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:44.848369 20438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:44.849500 20438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:44.849542 20438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:44.849568 20438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:40:44.850894 20438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:44.851136 20438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:44.851397 20438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:44.856026 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:44.856058 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.856134 20438 server_base.cc:1034] running on GCE node
W20260924 19:40:44.856251 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:44.856420 20438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:44.856637 20438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:44.857796 20438 hybrid_clock.cc:648] HybridClock initialized: now 1790278844857782 us; error 26 us; skew 500 ppm
I20260924 19:40:44.859084 20438 webserver.cc:460] Webserver started at http://127.0.98.3:42589/ using document root <none> and password file <none>
I20260924 19:40:44.859308 20438 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:44.859357 20438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:44.859480 20438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:44.860647 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e408cd2bbc3789d01e3b5b40e515076f"
server_key_iv: "84ec2187326bc69bde4d7f9e7e17987e"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.861063 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e408cd2bbc3789d01e3b5b40e515076f"
server_key_iv: "84ec2187326bc69bde4d7f9e7e17987e"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.862505 20438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:44.863348 20452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.863541 20438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:44.863659 20438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal
uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e"
format_stamp: "Formatted at 2026-09-24 19:40:44 on dist-test-slave-6kf3"
server_key: "e408cd2bbc3789d01e3b5b40e515076f"
server_key_iv: "84ec2187326bc69bde4d7f9e7e17987e"
server_key_version: "encryptionkey@0"
I20260924 19:40:44.863735 20438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:44.874312 20438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:44.874579 20438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:44.874680 20438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:44.874862 20438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:40:44.875164 20438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:40:44.875195 20438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.875222 20438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:40:44.875239 20438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:44.880764 20438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35549
I20260924 19:40:44.880811 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35549 every 8 connection(s)
I20260924 19:40:44.881108 20438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:40:44.884682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20438
I20260924 19:40:44.884790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:40:44.884961 392 external_mini_cluster.cc:1428] Setting key ce22e701961da3fa3411716acf3f2d45
I20260924 19:40:44.886198 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.881720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46393 (local address 127.0.98.62:39079)
0924 19:40:44.881768 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.881770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.881989 (+ 219us) server_negotiation.cc:407] Connection header received
0924 19:40:44.882482 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.882484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.882494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.882551 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.883185 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.883683 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.884696 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.884956 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.885013 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.885924 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.885925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.885926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:44.885930 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.885945 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:44.885949 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.885993 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.886082 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.886113 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:40:44.886373 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.881699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39079 (local address 127.0.98.3:46393)
0924 19:40:44.881871 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.881916 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.882376 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.882611 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.882614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.882622 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:40:44.883120 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.883125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.883741 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.883742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.884611 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.884614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.885078 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.885079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.885240 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.885771 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:40:44.885772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:40:44.885827 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:40:44.885854 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:40:44.885993 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:40:44.885998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:40:44.886001 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.886073 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.886175 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260924 19:40:44.886854 20566 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39079
I20260924 19:40:44.886941 20566 heartbeater.cc:464] Registering TS with master...
I20260924 19:40:44.887112 20566 heartbeater.cc:511] Master 127.0.98.62:39079 requested a full tablet report, sending...
I20260924 19:40:44.887415 20115 ts_manager.cc:194] Registered new tserver with Master: ee6e5b544a6b47d6ac686a7eb9251f7e (127.0.98.3:35549)
I20260924 19:40:44.887750 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46393
I20260924 19:40:44.896842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:40:44.900835 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.898130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.0.98.62:39079)
0924 19:40:44.898181 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.898183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.898266 (+ 83us) server_negotiation.cc:407] Connection header received
0924 19:40:44.898332 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.898332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.898345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.898392 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:44.898547 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.899047 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.899789 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.900024 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.900079 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.900608 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:44.900609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:44.900609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:44.900614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:44.900629 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:44.900632 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:44.900676 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.900735 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.900760 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:40:44.902344 20115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:40:44.903786 20115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:40:44.909842 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.905912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.98.1:44965)
0924 19:40:44.906102 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.906106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.906204 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:40:44.906256 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.906259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.906275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.906344 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.906567 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907432 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.908901 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.909466 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909536 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.909635 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.909706 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.909751 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260924 19:40:44.909827 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.905818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44965 (local address 127.0.0.1:55992)
0924 19:40:44.906143 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.906152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.906201 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.906422 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.906425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.906427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.906492 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.906494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907502 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.907503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.908811 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.908814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.909542 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.909615 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.909627 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.909694 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.909748 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":98,"threads_started":1}
I20260924 19:40:44.910213 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.906205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.0.98.3:35549)
0924 19:40:44.906362 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.906366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.906375 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:44.906409 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.906411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.906424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.906498 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.906722 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907848 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.908911 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.909580 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909647 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.909745 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.910092 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.910135 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260924 19:40:44.910238 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.905813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39197 (local address 127.0.0.1:35836)
0924 19:40:44.905967 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.906009 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.906104 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.906587 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.906588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.906590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.906636 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.906638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907899 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.907900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.909024 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.909027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.910033 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.910034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.910089 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.910093 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.910147 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.910183 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260924 19:40:44.910305 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.905979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.0.98.2:39197)
0924 19:40:44.906396 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:44.906400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:44.906408 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:40:44.906439 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:44.906440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:44.906452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:44.906517 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:44.906712 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907822 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909102 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.909870 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909924 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.910001 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:44.910182 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 19:40:44.910226 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":80,"threads_started":1}
I20260924 19:40:44.910023 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:44.906023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35549 (local address 127.0.0.1:36964)
0924 19:40:44.906223 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:44.906232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:44.906281 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:44.906613 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:44.906614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:44.906615 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:44.906645 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.906646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.907949 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.907950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.908839 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:44.908842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:44.909782 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:44.909783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:44.909843 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:44.909848 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:40:44.909898 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:40:44.909939 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
I20260924 19:40:44.910868 20369 tablet_service.cc:1467] Processing CreateTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 (DEFAULT_TABLE table=test-table [id=44625806606c449597c29e7af980afcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.910869 20238 tablet_service.cc:1467] Processing CreateTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 (DEFAULT_TABLE table=test-table [id=44625806606c449597c29e7af980afcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.911242 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424077ad9ef741708a227dce6d8ab2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.911242 20369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424077ad9ef741708a227dce6d8ab2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.911201 20500 tablet_service.cc:1467] Processing CreateTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 (DEFAULT_TABLE table=test-table [id=44625806606c449597c29e7af980afcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:40:44.911490 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424077ad9ef741708a227dce6d8ab2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:44.913952 20580 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Bootstrap starting.
I20260924 19:40:44.914450 20581 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Bootstrap starting.
I20260924 19:40:44.914677 20580 tablet_bootstrap.cc:654] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.915004 20580 log.cc:824] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.915076 20581 tablet_bootstrap.cc:654] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.915336 20581 log.cc:824] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.915640 20582 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Bootstrap starting.
I20260924 19:40:44.915733 20580 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: No bootstrap required, opened a new log
I20260924 19:40:44.915792 20580 ts_tablet_manager.cc:1397] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:44.915979 20581 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: No bootstrap required, opened a new log
I20260924 19:40:44.916051 20581 ts_tablet_manager.cc:1397] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:40:44.916260 20582 tablet_bootstrap.cc:654] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:44.916630 20582 log.cc:824] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:44.917374 20582 tablet_bootstrap.cc:492] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: No bootstrap required, opened a new log
I20260924 19:40:44.917438 20582 ts_tablet_manager.cc:1397] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:40:44.918104 20581 raft_consensus.cc:348] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.918231 20581 raft_consensus.cc:374] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.918260 20581 raft_consensus.cc:729] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd97e4c3ca64de4864f3ef87ddbc9d2, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.918236 20580 raft_consensus.cc:348] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.918387 20580 raft_consensus.cc:374] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.918385 20581 consensus_queue.cc:260] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.918426 20580 raft_consensus.cc:729] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6e5b544a6b47d6ac686a7eb9251f7e, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.918560 20580 consensus_queue.cc:260] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.918634 20581 ts_tablet_manager.cc:1428] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:40:44.918720 20435 heartbeater.cc:503] Master 127.0.98.62:39079 was elected leader, sending a full tablet report...
I20260924 19:40:44.918816 20580 ts_tablet_manager.cc:1428] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:40:44.918850 20566 heartbeater.cc:503] Master 127.0.98.62:39079 was elected leader, sending a full tablet report...
I20260924 19:40:44.919756 20582 raft_consensus.cc:348] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.919898 20582 raft_consensus.cc:374] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:44.919956 20582 raft_consensus.cc:729] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8135771d50b54522aa0c6b7eada62487, State: Initialized, Role: FOLLOWER
I20260924 19:40:44.920066 20582 consensus_queue.cc:260] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:44.920277 20582 ts_tablet_manager.cc:1428] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:40:44.920311 20304 heartbeater.cc:503] Master 127.0.98.62:39079 was elected leader, sending a full tablet report...
W20260924 19:40:45.014240 20436 tablet.cc:2369] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:45.131613 20567 tablet.cc:2369] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:40:45.147073 20305 tablet.cc:2369] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:40:45.226239 20586 raft_consensus.cc:482] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:40:45.226361 20586 raft_consensus.cc:504] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:45.226729 20586 leader_election.cc:288] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6e5b544a6b47d6ac686a7eb9251f7e (127.0.98.3:35549), 8135771d50b54522aa0c6b7eada62487 (127.0.98.1:44965)
I20260924 19:40:45.229332 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.226965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36809 (local address 127.0.98.1:44965)
0924 19:40:45.227004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:45.227007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:45.227018 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:40:45.227073 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:45.227074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:45.227085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:45.227147 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:45.227285 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.227761 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.228698 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.229040 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.229099 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:45.229109 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:45.229234 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:40:45.229267 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:40:45.229365 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.226913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44965 (local address 127.0.98.2:36809)
0924 19:40:45.226966 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:45.226977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:45.227019 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:45.227162 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:45.227163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:45.227165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:45.227224 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:45.227226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.227839 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.227840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:45.228608 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:45.228610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.229113 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.229114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:45.229174 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:45.229177 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:45.229220 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:40:45.229262 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:40:45.229372 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.226988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50605 (local address 127.0.98.3:35549)
0924 19:40:45.227032 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:45.227034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:45.227171 (+ 137us) server_negotiation.cc:407] Connection header received
0924 19:40:45.227220 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:45.227222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:45.227230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:45.227273 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:40:45.227399 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.227848 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.228749 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.229085 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.229143 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:45.229152 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:45.229271 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:40:45.229305 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:40:45.229390 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.226920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35549 (local address 127.0.98.2:50605)
0924 19:40:45.227122 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:45.227129 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:40:45.227165 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:40:45.227284 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:40:45.227284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:40:45.227286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:40:45.227349 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:45.227351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.227921 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.227922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:45.228671 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:40:45.228674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.229151 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.229151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:40:45.229217 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:45.229220 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:40:45.229262 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:40:45.229298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":22,"thread_start_us":130,"threads_started":1}
I20260924 19:40:45.229687 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424077ad9ef741708a227dce6d8ab2c1" candidate_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8135771d50b54522aa0c6b7eada62487" is_pre_election: true
I20260924 19:40:45.229696 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424077ad9ef741708a227dce6d8ab2c1" candidate_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" is_pre_election: true
I20260924 19:40:45.229825 20520 raft_consensus.cc:2454] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd97e4c3ca64de4864f3ef87ddbc9d2 in term 0.
I20260924 19:40:45.229825 20258 raft_consensus.cc:2454] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd97e4c3ca64de4864f3ef87ddbc9d2 in term 0.
I20260924 19:40:45.230031 20323 leader_election.cc:302] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd97e4c3ca64de4864f3ef87ddbc9d2, ee6e5b544a6b47d6ac686a7eb9251f7e; no voters:
I20260924 19:40:45.230157 20586 raft_consensus.cc:2781] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:40:45.230219 20586 raft_consensus.cc:482] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:40:45.230242 20586 raft_consensus.cc:3037] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:45.230918 20586 raft_consensus.cc:504] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:45.231036 20586 leader_election.cc:288] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [CANDIDATE]: Term 1 election: Requested vote from peers ee6e5b544a6b47d6ac686a7eb9251f7e (127.0.98.3:35549), 8135771d50b54522aa0c6b7eada62487 (127.0.98.1:44965)
I20260924 19:40:45.231196 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424077ad9ef741708a227dce6d8ab2c1" candidate_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8135771d50b54522aa0c6b7eada62487"
I20260924 19:40:45.231212 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424077ad9ef741708a227dce6d8ab2c1" candidate_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e"
I20260924 19:40:45.231281 20258 raft_consensus.cc:3037] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:45.231287 20520 raft_consensus.cc:3037] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:45.232160 20520 raft_consensus.cc:2454] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd97e4c3ca64de4864f3ef87ddbc9d2 in term 1.
I20260924 19:40:45.232160 20258 raft_consensus.cc:2454] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd97e4c3ca64de4864f3ef87ddbc9d2 in term 1.
I20260924 19:40:45.232306 20323 leader_election.cc:302] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd97e4c3ca64de4864f3ef87ddbc9d2, ee6e5b544a6b47d6ac686a7eb9251f7e; no voters:
I20260924 19:40:45.232388 20586 raft_consensus.cc:2781] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:45.232510 20586 raft_consensus.cc:686] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd97e4c3ca64de4864f3ef87ddbc9d2, State: Running, Role: LEADER
I20260924 19:40:45.232599 20586 consensus_queue.cc:237] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } }
I20260924 19:40:45.233335 20116 catalog_manager.cc:5495] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd97e4c3ca64de4864f3ef87ddbc9d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd97e4c3ca64de4864f3ef87ddbc9d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 } health_report { overall_health: UNKNOWN } } }
I20260924 19:40:45.263720 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.261718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.0.98.2:39197)
0924 19:40:45.261764 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:45.261765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:45.261793 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:40:45.261827 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:45.261828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:45.261844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:45.261897 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:40:45.261993 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.262445 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.263127 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:45.263308 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:45.263355 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:45.263444 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:40:45.263562 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:40:45.263615 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:45.263631 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 19:40:45.263646 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:40:45.265343 20258 raft_consensus.cc:1261] T 424077ad9ef741708a227dce6d8ab2c1 P 8135771d50b54522aa0c6b7eada62487 [term 1 FOLLOWER]: Refusing update from remote peer 4dd97e4c3ca64de4864f3ef87ddbc9d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:45.265344 20520 raft_consensus.cc:1261] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 1 FOLLOWER]: Refusing update from remote peer 4dd97e4c3ca64de4864f3ef87ddbc9d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:40:45.265659 20590 consensus_queue.cc:1035] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8135771d50b54522aa0c6b7eada62487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:45.265782 20586 consensus_queue.cc:1035] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6e5b544a6b47d6ac686a7eb9251f7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:40:45.268536 20595 mvcc.cc:205] Tried to move back new op lower bound from 7332982150204628992 to 7332982150073073664. Current Snapshot: MvccSnapshot[applied={T|T < 7332982150204628992}]
I20260924 19:40:45.270057 20115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:45.270149 20115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:40:45.270823 20115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b2cd5b4d434d51bbee1bafd9fc6ec6: Sending DeleteTablet for 3 replicas of tablet 424077ad9ef741708a227dce6d8ab2c1
I20260924 19:40:45.271144 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:45 UTC) from {username='slave'} at 127.0.0.1:35836
I20260924 19:40:45.271152 20500 tablet_service.cc:1514] Processing DeleteTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:45 UTC) from {username='slave'} at 127.0.0.1:36964
I20260924 19:40:45.271339 20604 tablet_replica.cc:330] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: stopping tablet replica
I20260924 19:40:45.271330 20238 tablet_service.cc:1514] Processing DeleteTablet for tablet 424077ad9ef741708a227dce6d8ab2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:40:45 UTC) from {username='slave'} at 127.0.0.1:55992
I20260924 19:40:45.271425 20604 raft_consensus.cc:2229] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:40:45.271471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20176
I20260924 19:40:45.271518 20604 raft_consensus.cc:2258] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:45.272928 20604 ts_tablet_manager.cc:1905] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:45.274135 20603 tablet_replica.cc:330] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: stopping tablet replica
I20260924 19:40:45.274232 20603 raft_consensus.cc:2229] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:40:45.274320 20603 raft_consensus.cc:2258] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:40:45.274739 20603 ts_tablet_manager.cc:1905] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:40:45.276057 20604 ts_tablet_manager.cc:1918] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:45.276175 20604 log.cc:1196] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-2/wal/wals/424077ad9ef741708a227dce6d8ab2c1
I20260924 19:40:45.276188 20603 ts_tablet_manager.cc:1918] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:40:45.276250 20603 log.cc:1196] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Deleting WAL directory at /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278584646480-392-0/minicluster-data/ts-1/wal/wals/424077ad9ef741708a227dce6d8ab2c1
I20260924 19:40:45.276465 20603 ts_tablet_manager.cc:1939] T 424077ad9ef741708a227dce6d8ab2c1 P 4dd97e4c3ca64de4864f3ef87ddbc9d2: Deleting consensus metadata
I20260924 19:40:45.276610 20604 ts_tablet_manager.cc:1939] T 424077ad9ef741708a227dce6d8ab2c1 P ee6e5b544a6b47d6ac686a7eb9251f7e: Deleting consensus metadata
I20260924 19:40:45.276818 20101 catalog_manager.cc:4841] TS 4dd97e4c3ca64de4864f3ef87ddbc9d2 (127.0.98.2:39197): tablet 424077ad9ef741708a227dce6d8ab2c1 (table test-table [id=44625806606c449597c29e7af980afcb]) successfully deleted
I20260924 19:40:45.277189 20101 catalog_manager.cc:4841] TS ee6e5b544a6b47d6ac686a7eb9251f7e (127.0.98.3:35549): tablet 424077ad9ef741708a227dce6d8ab2c1 (table test-table [id=44625806606c449597c29e7af980afcb]) successfully deleted
W20260924 19:40:45.277901 20103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44965 (error 108)
I20260924 19:40:45.278096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20307
I20260924 19:40:45.282778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20438
I20260924 19:40:45.282924 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:45.282795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44965 (local address 127.0.0.1:55996)
0924 19:40:45.282848 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:40:45.282891 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:40:45.286144 20103 catalog_manager.cc:4584] TS 8135771d50b54522aa0c6b7eada62487 (127.0.98.1:44965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 424077ad9ef741708a227dce6d8ab2c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44965: connect: Connection refused (error 111)
I20260924 19:40:45.286382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20085
2026-09-24T19:40:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (919 ms)
[----------] 6 tests from EncryptionPolicyTest (8195 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:45 dist-test-slave-6kf3 krb5kdc[20609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:45 dist-test-slave-6kf3 krb5kdc[20609](info): set up 2 sockets
Sep 24 19:40:45 dist-test-slave-6kf3 krb5kdc[20609](info): commencing operation
krb5kdc: starting...
W20260924 19:40:46.235667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:46.344621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:46.424355 20622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:46.424557 20622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:46.424578 20622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:46.425686 20622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:46.425727 20622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:46.425740 20622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:46.425750 20622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:46.425760 20622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:46.427212 20622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:46.427438 20622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:46.427641 20622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:46.432199 20628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:46.432229 20627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:46.432250 20622 server_base.cc:1034] running on GCE node
W20260924 19:40:46.432220 20630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:46.432615 20622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:46.432893 20622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:46.434020 20622 hybrid_clock.cc:648] HybridClock initialized: now 1790278846434011 us; error 28 us; skew 500 ppm
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:46.439795 20622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:46.440245 20622 webserver.cc:460] Webserver started at http://127.0.0.1:34077/ using document root <none> and password file <none>
I20260924 19:40:46.440435 20622 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:46.440475 20622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:46.440564 20622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:46.441570 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "552bffb6f8af4ed8b1ef986fbf6956e2"
format_stamp: "Formatted at 2026-09-24 19:40:46 on dist-test-slave-6kf3"
server_key: "bd34a3207e5a6cf29948a4e36d48049a"
server_key_iv: "c5746778c81cd4e93ca1f675175d9149"
server_key_version: "encryptionkey@0"
I20260924 19:40:46.441849 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "552bffb6f8af4ed8b1ef986fbf6956e2"
format_stamp: "Formatted at 2026-09-24 19:40:46 on dist-test-slave-6kf3"
server_key: "bd34a3207e5a6cf29948a4e36d48049a"
server_key_iv: "c5746778c81cd4e93ca1f675175d9149"
server_key_version: "encryptionkey@0"
I20260924 19:40:46.443010 20622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:40:46.443699 20637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:46.443876 20622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:40:46.443982 20622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "552bffb6f8af4ed8b1ef986fbf6956e2"
format_stamp: "Formatted at 2026-09-24 19:40:46 on dist-test-slave-6kf3"
server_key: "bd34a3207e5a6cf29948a4e36d48049a"
server_key_iv: "c5746778c81cd4e93ca1f675175d9149"
server_key_version: "encryptionkey@0"
I20260924 19:40:46.444054 20622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:46.459146 20622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:46.460989 20622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:46.461100 20622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:46.464555 20622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44333
I20260924 19:40:46.464589 20689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44333 every 8 connection(s)
I20260924 19:40:46.464891 20622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:46.465687 20690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:46.467785 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: Bootstrap starting.
I20260924 19:40:46.468303 20690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:46.468513 20690 log.cc:824] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:46.469096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20622
I20260924 19:40:46.469116 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: No bootstrap required, opened a new log
I20260924 19:40:46.469197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:46.469362 392 external_mini_cluster.cc:1428] Setting key 971e890a547046d8b3628ec947622eb0
I20260924 19:40:46.471064 20690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260924 19:40:46.471227 20690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:46.471262 20690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552bffb6f8af4ed8b1ef986fbf6956e2, State: Initialized, Role: FOLLOWER
I20260924 19:40:46.471377 20690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260924 19:40:46.471448 20690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:46.471468 20690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:46.471514 20690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:46.472499 20690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260924 19:40:46.472610 20690 leader_election.cc:302] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 552bffb6f8af4ed8b1ef986fbf6956e2; no voters:
I20260924 19:40:46.472748 20690 leader_election.cc:288] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:46.472827 20695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:46.472980 20690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:46.472983 20695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [term 1 LEADER]: Becoming Leader. State: Replica: 552bffb6f8af4ed8b1ef986fbf6956e2, State: Running, Role: LEADER
I20260924 19:40:46.473088 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260924 19:40:46.473476 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 552bffb6f8af4ed8b1ef986fbf6956e2. Latest consensus state: current_term: 1 leader_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260924 19:40:46.473556 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:46.473698 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552bffb6f8af4ed8b1ef986fbf6956e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260924 19:40:46.473747 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2 [sys.catalog]: This master's current role is: LEADER
W20260924 19:40:46.475273 20711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:40:46.475325 20711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:40:46.475378 20704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:46.475543 20704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:40:46.477411 20704 catalog_manager.cc:1347] Generated new cluster ID: 8931537aa73a4f5b97b1622dac82f696
I20260924 19:40:46.477470 20704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:46.481065 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:46.469862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47197 (local address 127.0.0.1:44333)
0924 19:40:46.470043 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:46.470052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:46.470096 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:40:46.471030 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:46.471039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:46.471109 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:46.471255 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:46.471907 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:46.473298 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:46.474284 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:46.474784 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:46.474861 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:46.477392 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:46.477395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:46.477396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:46.477407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:46.479228 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:46.479616 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:46.479619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:46.479619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:46.479733 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:46.479924 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:46.479924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:46.479925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:46.480221 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:46.480362 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:46.480785 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 19:40:46.480881 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260924 19:40:46.481875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:46.490024 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:46.483385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48735 (local address 127.0.0.1:44333)
0924 19:40:46.483415 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:46.483417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:46.483424 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:40:46.483462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:46.483463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:46.483471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:46.483511 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:46.484149 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:46.484861 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:46.485849 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:46.486159 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:46.486209 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:46.487493 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:46.487495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:46.487495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:46.487501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:46.488956 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:46.489330 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:46.489331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:46.489332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:46.489386 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:46.489569 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:46.489569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:46.489570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:46.489689 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:46.489792 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:46.489876 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:40:46.489936 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:40:46.491802 20704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:46.492314 20704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:46.499960 20704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 552bffb6f8af4ed8b1ef986fbf6956e2: Generated new TSK 0
I20260924 19:40:46.500133 20704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:46.503019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20622
W20260924 19:40:46.506736 20607 connection.cc:535] client connection to 127.0.0.1:44333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44333): Connection reset by peer (error 104)
2026-09-24T19:40:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20721](info): set up 2 sockets
Sep 24 19:40:46 dist-test-slave-6kf3 krb5kdc[20721](info): commencing operation
krb5kdc: starting...
W20260924 19:40:47.447566 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:47 dist-test-slave-6kf3 krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:47.556535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:47.634598 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:47.634799 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:47.634821 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:47.635919 20734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:47.636003 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:47.636018 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:47.636029 20734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:47.636039 20734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:47.637476 20734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:47.637662 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:47.637828 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:47.642347 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:47.642374 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:47.642354 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:47.642659 20734 server_base.cc:1034] running on GCE node
I20260924 19:40:47.642889 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:47.643123 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:47.644264 20734 hybrid_clock.cc:648] HybridClock initialized: now 1790278847644254 us; error 28 us; skew 500 ppm
Sep 24 19:40:47 dist-test-slave-6kf3 krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:47.646546 20734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:47.646955 20734 webserver.cc:460] Webserver started at http://127.0.0.1:35023/ using document root <none> and password file <none>
I20260924 19:40:47.647149 20734 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:47.647194 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:47.647312 20734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:47.648350 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "46d5fca6aef547a0addfb14c9327cf28"
format_stamp: "Formatted at 2026-09-24 19:40:47 on dist-test-slave-6kf3"
server_key: "6b022c7b84a8395c7b6687cc7f2255eb"
server_key_iv: "20ae2e1ec6558b5b391e9f19fff66fc7"
server_key_version: "encryptionkey@0"
I20260924 19:40:47.648665 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "46d5fca6aef547a0addfb14c9327cf28"
format_stamp: "Formatted at 2026-09-24 19:40:47 on dist-test-slave-6kf3"
server_key: "6b022c7b84a8395c7b6687cc7f2255eb"
server_key_iv: "20ae2e1ec6558b5b391e9f19fff66fc7"
server_key_version: "encryptionkey@0"
I20260924 19:40:47.649812 20734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:47.650414 20749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:47.650565 20734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:47.650658 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "46d5fca6aef547a0addfb14c9327cf28"
format_stamp: "Formatted at 2026-09-24 19:40:47 on dist-test-slave-6kf3"
server_key: "6b022c7b84a8395c7b6687cc7f2255eb"
server_key_iv: "20ae2e1ec6558b5b391e9f19fff66fc7"
server_key_version: "encryptionkey@0"
I20260924 19:40:47.650715 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:47.665247 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:47.667273 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:47.667405 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:47.671130 20734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39225
I20260924 19:40:47.671164 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39225 every 8 connection(s)
I20260924 19:40:47.671483 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:47.672336 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:47.674583 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28: Bootstrap starting.
I20260924 19:40:47.675127 20802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:47.675395 20802 log.cc:824] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:47.676275 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28: No bootstrap required, opened a new log
I20260924 19:40:47.677835 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260924 19:40:47.677981 20802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:47.678004 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d5fca6aef547a0addfb14c9327cf28, State: Initialized, Role: FOLLOWER
I20260924 19:40:47.678141 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260924 19:40:47.678201 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:47.678228 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:47.678273 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:47.678781 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260924 19:40:47.678875 20802 leader_election.cc:302] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d5fca6aef547a0addfb14c9327cf28; no voters:
I20260924 19:40:47.679044 20802 leader_election.cc:288] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:47.679080 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:47.679219 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [term 1 LEADER]: Becoming Leader. State: Replica: 46d5fca6aef547a0addfb14c9327cf28, State: Running, Role: LEADER
I20260924 19:40:47.679277 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:47.679313 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260924 19:40:47.679670 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d5fca6aef547a0addfb14c9327cf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260924 19:40:47.679682 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d5fca6aef547a0addfb14c9327cf28. Latest consensus state: current_term: 1 leader_uuid: "46d5fca6aef547a0addfb14c9327cf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5fca6aef547a0addfb14c9327cf28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260924 19:40:47.679822 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:47.679843 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:47.680238 20811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:47.680677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20734
I20260924 19:40:47.680781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:47.680928 20811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:47.680953 392 external_mini_cluster.cc:1428] Setting key 41280651ae821376514cade655087fc1
I20260924 19:40:47.682427 20811 catalog_manager.cc:1347] Generated new cluster ID: a320f0efe86f4cf190fcd546884d9ab4
I20260924 19:40:47.682488 20811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:47.688068 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:47.681361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.0.0.1:39225)
0924 19:40:47.684085 (+ 2724us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:47.684094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:47.684120 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:40:47.684216 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:47.684222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:47.684270 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:47.684388 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:47.684606 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:47.685505 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:47.686288 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:47.686701 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:47.686793 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:47.687441 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:47.687445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:47.687446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:47.687619 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:47.687650 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:47.687662 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:47.687728 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:47.687818 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:40:47.687844 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2675,"thread_start_us":57,"threads_started":1}
I20260924 19:40:47.688951 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:47.692898 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:47.690217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58047 (local address 127.0.0.1:39225)
0924 19:40:47.690251 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:47.690253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:47.690360 (+ 107us) server_negotiation.cc:407] Connection header received
0924 19:40:47.690410 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:47.690411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:47.690421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:47.690465 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:47.690589 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:47.691066 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:47.691760 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:47.692038 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:47.692100 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:47.692678 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:47.692679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:47.692680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:47.692684 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:47.692697 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:47.692702 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:47.692745 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:47.692811 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:40:47.692821 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260924 19:40:47.703296 20811 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:47.703861 20811 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:47.709496 20811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d5fca6aef547a0addfb14c9327cf28: Generated new TSK 0
I20260924 19:40:47.709734 20811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:47.711817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20734
W20260924 19:40:47.715466 20719 connection.cc:535] client connection to 127.0.0.1:39225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39225): Connection reset by peer (error 104)
2026-09-24T19:40:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:47 dist-test-slave-6kf3 krb5kdc[20833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:47 dist-test-slave-6kf3 krb5kdc[20833](info): set up 2 sockets
Sep 24 19:40:47 dist-test-slave-6kf3 krb5kdc[20833](info): commencing operation
krb5kdc: starting...
W20260924 19:40:48.652180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:48 dist-test-slave-6kf3 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:48.759661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:48.835873 20846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:48.836097 20846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:48.836126 20846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:48.837257 20846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:48.837301 20846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:48.837319 20846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:48.837335 20846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:48.837352 20846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:48.838807 20846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:48.839054 20846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:48.839253 20846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:48.843679 20854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:48.843685 20851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:48.843823 20846 server_base.cc:1034] running on GCE node
W20260924 19:40:48.843889 20852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:48.844144 20846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:48.844394 20846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:48.845530 20846 hybrid_clock.cc:648] HybridClock initialized: now 1790278848845509 us; error 32 us; skew 500 ppm
Sep 24 19:40:48 dist-test-slave-6kf3 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:48.847743 20846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:48.848197 20846 webserver.cc:460] Webserver started at http://127.0.0.1:46601/ using document root <none> and password file <none>
I20260924 19:40:48.848389 20846 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:48.848435 20846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:48.848553 20846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:48.849531 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "4c4e7f66883a46c2b077e7d582738f61"
format_stamp: "Formatted at 2026-09-24 19:40:48 on dist-test-slave-6kf3"
server_key: "34400ee7b078d00b1e21218b2285c32d"
server_key_iv: "6f63069d4d1d3c354f2bb8946fdda5fc"
server_key_version: "encryptionkey@0"
I20260924 19:40:48.849846 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c4e7f66883a46c2b077e7d582738f61"
format_stamp: "Formatted at 2026-09-24 19:40:48 on dist-test-slave-6kf3"
server_key: "34400ee7b078d00b1e21218b2285c32d"
server_key_iv: "6f63069d4d1d3c354f2bb8946fdda5fc"
server_key_version: "encryptionkey@0"
I20260924 19:40:48.851004 20846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:48.851640 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:48.851785 20846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:48.851864 20846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "4c4e7f66883a46c2b077e7d582738f61"
format_stamp: "Formatted at 2026-09-24 19:40:48 on dist-test-slave-6kf3"
server_key: "34400ee7b078d00b1e21218b2285c32d"
server_key_iv: "6f63069d4d1d3c354f2bb8946fdda5fc"
server_key_version: "encryptionkey@0"
I20260924 19:40:48.851922 20846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:48.855083 20846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:48.856940 20846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:48.857072 20846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:48.861097 20846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44257
I20260924 19:40:48.861131 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44257 every 8 connection(s)
I20260924 19:40:48.861445 20846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:48.862362 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:48.863191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20846
I20260924 19:40:48.863303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:48.863474 392 external_mini_cluster.cc:1428] Setting key 1e6a24cd9a52fa21340b0ba108afe907
I20260924 19:40:48.864748 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61: Bootstrap starting.
I20260924 19:40:48.865396 20914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:48.865676 20914 log.cc:824] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:48.865746 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:48.864023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58009 (local address 127.0.0.1:44257)
0924 19:40:48.864191 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:48.864199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:48.864223 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:40:48.864627 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:48.864634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:48.864660 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:48.864777 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:48.865360 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:48.865363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:48.865363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:48.865366 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:48.865400 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:48.865410 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:48.865499 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:48.865533 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:40:48.865565 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 19:40:48.866322 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61: No bootstrap required, opened a new log
I20260924 19:40:48.866613 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:48.868700 20914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260924 19:40:48.868875 20914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:48.868920 20914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4e7f66883a46c2b077e7d582738f61, State: Initialized, Role: FOLLOWER
I20260924 19:40:48.869050 20914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260924 19:40:48.869141 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:48.868077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36719 (local address 127.0.0.1:44257)
0924 19:40:48.868125 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:48.868127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:48.868261 (+ 134us) server_negotiation.cc:407] Connection header received
0924 19:40:48.868322 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:48.868322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:48.868326 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:48.868366 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:48.868856 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:48.868856 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:48.868857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:48.868860 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:48.868872 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:48.868875 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:48.868928 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:48.869051 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:40:48.869064 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:40:48.869143 20914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:48.869210 20914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:48.869252 20914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:48.869822 20914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260924 19:40:48.869925 20914 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4e7f66883a46c2b077e7d582738f61; no voters:
I20260924 19:40:48.870087 20914 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:48.870126 20924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:48.870304 20914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:48.870301 20924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [term 1 LEADER]: Becoming Leader. State: Replica: 4c4e7f66883a46c2b077e7d582738f61, State: Running, Role: LEADER
I20260924 19:40:48.870472 20924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260924 19:40:48.870828 20926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4e7f66883a46c2b077e7d582738f61. Latest consensus state: current_term: 1 leader_uuid: "4c4e7f66883a46c2b077e7d582738f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260924 19:40:48.870836 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4e7f66883a46c2b077e7d582738f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4e7f66883a46c2b077e7d582738f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260924 19:40:48.870992 20926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:48.871001 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:48.871411 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:48.871830 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:48.873457 20933 catalog_manager.cc:1347] Generated new cluster ID: 92f08bd69be642049ad762c65edc3a66
I20260924 19:40:48.873507 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:48.891810 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:48.892040 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:48.897181 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4e7f66883a46c2b077e7d582738f61: Generated new TSK 0
I20260924 19:40:48.897360 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:48.908567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20846
2026-09-24T19:40:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:48 dist-test-slave-6kf3 krb5kdc[20945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:48 dist-test-slave-6kf3 krb5kdc[20945](info): set up 2 sockets
Sep 24 19:40:48 dist-test-slave-6kf3 krb5kdc[20945](info): commencing operation
krb5kdc: starting...
W20260924 19:40:49.847582 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:49 dist-test-slave-6kf3 krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:49.953049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:50.031872 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:50.032117 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:50.032151 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:50.033288 20958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:50.033340 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:50.033362 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:50.033381 20958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:50.033401 20958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:50.034883 20958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:50.035143 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:50.035369 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:50.040004 20964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:50.040042 20963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:50.040010 20966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:50.040175 20958 server_base.cc:1034] running on GCE node
I20260924 19:40:50.040477 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:50.040732 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:50.041860 20958 hybrid_clock.cc:648] HybridClock initialized: now 1790278850041833 us; error 44 us; skew 500 ppm
Sep 24 19:40:50 dist-test-slave-6kf3 krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:50.044153 20958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:50.044559 20958 webserver.cc:460] Webserver started at http://127.0.0.1:43081/ using document root <none> and password file <none>
I20260924 19:40:50.044747 20958 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:50.044788 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:50.044902 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:50.045920 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "fdaffe91ec884eabb21689433ae470bc"
format_stamp: "Formatted at 2026-09-24 19:40:50 on dist-test-slave-6kf3"
server_key: "40e1eebdb38afc9775dad875b6097df7"
server_key_iv: "bd9709b9262387650bcce37fd37dd5f4"
server_key_version: "encryptionkey@0"
I20260924 19:40:50.046238 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdaffe91ec884eabb21689433ae470bc"
format_stamp: "Formatted at 2026-09-24 19:40:50 on dist-test-slave-6kf3"
server_key: "40e1eebdb38afc9775dad875b6097df7"
server_key_iv: "bd9709b9262387650bcce37fd37dd5f4"
server_key_version: "encryptionkey@0"
I20260924 19:40:50.047485 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:50.048175 20973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:50.048334 20958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:50.048416 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "fdaffe91ec884eabb21689433ae470bc"
format_stamp: "Formatted at 2026-09-24 19:40:50 on dist-test-slave-6kf3"
server_key: "40e1eebdb38afc9775dad875b6097df7"
server_key_iv: "bd9709b9262387650bcce37fd37dd5f4"
server_key_version: "encryptionkey@0"
I20260924 19:40:50.048470 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:50.076102 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:50.077862 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:50.077970 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:50.081504 20958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37367
I20260924 19:40:50.081555 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37367 every 8 connection(s)
I20260924 19:40:50.081866 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:50.082696 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:50.084919 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc: Bootstrap starting.
I20260924 19:40:50.085448 21026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:50.085697 21026 log.cc:824] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:50.086324 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc: No bootstrap required, opened a new log
I20260924 19:40:50.087198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 20958
I20260924 19:40:50.087299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:50.087455 392 external_mini_cluster.cc:1428] Setting key 6acbc49799a0d6bd5ff0f25f9c2357dd
I20260924 19:40:50.088080 21026 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260924 19:40:50.088218 21026 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:50.088249 21026 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdaffe91ec884eabb21689433ae470bc, State: Initialized, Role: FOLLOWER
I20260924 19:40:50.088367 21026 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260924 19:40:50.088433 21026 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:50.088462 21026 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:50.088553 21026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:50.089506 21026 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260924 19:40:50.089610 21026 leader_election.cc:302] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdaffe91ec884eabb21689433ae470bc; no voters:
I20260924 19:40:50.089754 21026 leader_election.cc:288] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:50.089793 21031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:50.089933 21031 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [term 1 LEADER]: Becoming Leader. State: Replica: fdaffe91ec884eabb21689433ae470bc, State: Running, Role: LEADER
I20260924 19:40:50.089999 21026 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:50.090111 21031 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260924 19:40:50.090853 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdaffe91ec884eabb21689433ae470bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260924 19:40:50.090863 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdaffe91ec884eabb21689433ae470bc. Latest consensus state: current_term: 1 leader_uuid: "fdaffe91ec884eabb21689433ae470bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaffe91ec884eabb21689433ae470bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260924 19:40:50.091065 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:50.090993 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:50.091241 21044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:50.091903 21044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:50.093751 21044 catalog_manager.cc:1347] Generated new cluster ID: bd3d8923ad9b416f8f80d8c017eda05d
Sep 24 19:40:50 dist-test-slave-6kf3 krb5kdc[20945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:40:50.093796 21044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:50.097028 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:50.087921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42293 (local address 127.0.0.1:37367)
0924 19:40:50.088121 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:50.088129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:50.088153 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:40:50.088245 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:50.088251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:50.088307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:50.088440 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:50.089197 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:50.090345 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:50.091452 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:50.091994 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:50.092094 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:50.094219 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:50.094223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:50.094224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:50.094236 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:50.095643 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:50.095968 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:50.095971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:50.095972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:50.096083 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:50.096255 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:50.096256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:50.096257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:50.096476 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:50.096621 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:50.096699 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:40:50.096826 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260924 19:40:50.097905 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:50.104432 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:50.099147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48431 (local address 127.0.0.1:37367)
0924 19:40:50.099180 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:50.099182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:50.099273 (+ 91us) server_negotiation.cc:407] Connection header received
0924 19:40:50.099320 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:50.099321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:50.099330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:50.099379 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:50.099836 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:50.100347 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:50.100994 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:50.101206 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:50.101256 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:50.102261 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:50.102262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:50.102262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:50.102281 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:50.103464 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:50.103775 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:50.103776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:50.103777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:50.103827 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:50.103995 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:50.103996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:50.103996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:50.104103 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:50.104184 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:50.104283 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:40:50.104350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:40:50.105273 21044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:50.105911 21044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:50.112311 21044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdaffe91ec884eabb21689433ae470bc: Generated new TSK 0
I20260924 19:40:50.112489 21044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:50.118829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 20958
2026-09-24T19:40:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:50 dist-test-slave-6kf3 krb5kdc[21057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:50 dist-test-slave-6kf3 krb5kdc[21057](info): set up 2 sockets
Sep 24 19:40:50 dist-test-slave-6kf3 krb5kdc[21057](info): commencing operation
krb5kdc: starting...
W20260924 19:40:51.067618 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:51 dist-test-slave-6kf3 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:51.178206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:51.258791 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:51.259035 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:51.259069 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:51.260278 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:51.260325 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:51.260346 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:51.260365 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:51.260385 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:51.261916 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:51.262184 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:51.262399 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:51.267122 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:51.267087 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:51.267081 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:51.267328 21070 server_base.cc:1034] running on GCE node
I20260924 19:40:51.267530 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:51.267786 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:51.268929 21070 hybrid_clock.cc:648] HybridClock initialized: now 1790278851268913 us; error 29 us; skew 500 ppm
Sep 24 19:40:51 dist-test-slave-6kf3 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:51.271301 21070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:51.271701 21070 webserver.cc:460] Webserver started at http://127.0.0.1:43751/ using document root <none> and password file <none>
I20260924 19:40:51.271899 21070 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:51.271970 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:51.272082 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:51.273114 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "be6d4c952bc543e59937ab42297bfe16"
format_stamp: "Formatted at 2026-09-24 19:40:51 on dist-test-slave-6kf3"
server_key: "2de214ab2aa243224a081a89f2a7ca61"
server_key_iv: "9dfdd286592071b57d7ccfc4b08a004c"
server_key_version: "encryptionkey@0"
I20260924 19:40:51.273429 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "be6d4c952bc543e59937ab42297bfe16"
format_stamp: "Formatted at 2026-09-24 19:40:51 on dist-test-slave-6kf3"
server_key: "2de214ab2aa243224a081a89f2a7ca61"
server_key_iv: "9dfdd286592071b57d7ccfc4b08a004c"
server_key_version: "encryptionkey@0"
I20260924 19:40:51.274648 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:40:51.275326 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:51.275467 21070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:40:51.275547 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "be6d4c952bc543e59937ab42297bfe16"
format_stamp: "Formatted at 2026-09-24 19:40:51 on dist-test-slave-6kf3"
server_key: "2de214ab2aa243224a081a89f2a7ca61"
server_key_iv: "9dfdd286592071b57d7ccfc4b08a004c"
server_key_version: "encryptionkey@0"
I20260924 19:40:51.275607 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:51.294896 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:51.297072 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:51.297221 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:51.300974 21070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41781
I20260924 19:40:51.301117 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41781 every 8 connection(s)
I20260924 19:40:51.301360 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:51.302184 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:51.302815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21070
I20260924 19:40:51.302932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:51.303175 392 external_mini_cluster.cc:1428] Setting key 07c83e8100886908602230a3d88de04b
I20260924 19:40:51.305015 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16: Bootstrap starting.
I20260924 19:40:51.305590 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:51.305882 21138 log.cc:824] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:51.306581 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16: No bootstrap required, opened a new log
I20260924 19:40:51.308349 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:51.303679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49733 (local address 127.0.0.1:41781)
0924 19:40:51.303831 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:51.303838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:51.303863 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:40:51.304269 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:51.304276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:51.304323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:51.304434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:51.304682 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:51.305589 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:51.306634 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:51.307034 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:51.307129 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:51.307743 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:51.307747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:51.307747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:51.307894 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:51.307923 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:51.307958 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:51.308002 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:51.308089 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:40:51.308150 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 19:40:51.308719 21138 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260924 19:40:51.308878 21138 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:51.308921 21138 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d4c952bc543e59937ab42297bfe16, State: Initialized, Role: FOLLOWER
I20260924 19:40:51.309036 21138 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260924 19:40:51.309118 21138 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:51.309154 21138 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:51.309201 21138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:51.309245 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:51.309866 21138 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260924 19:40:51.309978 21138 leader_election.cc:302] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6d4c952bc543e59937ab42297bfe16; no voters:
I20260924 19:40:51.310154 21138 leader_election.cc:288] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:51.310200 21143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:51.310391 21143 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [term 1 LEADER]: Becoming Leader. State: Replica: be6d4c952bc543e59937ab42297bfe16, State: Running, Role: LEADER
I20260924 19:40:51.310475 21138 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:51.310519 21143 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260924 19:40:51.310916 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6d4c952bc543e59937ab42297bfe16. Latest consensus state: current_term: 1 leader_uuid: "be6d4c952bc543e59937ab42297bfe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } } }
I20260924 19:40:51.311003 21147 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6d4c952bc543e59937ab42297bfe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6d4c952bc543e59937ab42297bfe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } } }
I20260924 19:40:51.311035 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:51.311116 21147 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:51.311331 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:51.312026 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:51.313645 21157 catalog_manager.cc:1347] Generated new cluster ID: a065f69a200946218475d4d7b83fdbc5
I20260924 19:40:51.313695 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:51.314284 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:51.310645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57353 (local address 127.0.0.1:41781)
0924 19:40:51.310679 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:51.310681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:51.311078 (+ 397us) server_negotiation.cc:407] Connection header received
0924 19:40:51.311192 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:51.311193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:51.311204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:51.311231 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:51.311410 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:51.312170 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:51.313128 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:51.313378 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:51.313437 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:51.314065 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:51.314067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:51.314067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:51.314071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:51.314087 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:51.314091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:51.314142 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:51.314195 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:40:51.314217 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:40:51.323993 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:51.324536 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:51.327955 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6d4c952bc543e59937ab42297bfe16: Generated new TSK 0
I20260924 19:40:51.328140 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:51.336839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21070
2026-09-24T19:40:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:51 dist-test-slave-6kf3 krb5kdc[21169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:51 dist-test-slave-6kf3 krb5kdc[21169](info): set up 2 sockets
Sep 24 19:40:51 dist-test-slave-6kf3 krb5kdc[21169](info): commencing operation
krb5kdc: starting...
W20260924 19:40:52.279738 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:52 dist-test-slave-6kf3 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:52.390851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:52.472131 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:52.472333 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:52.472363 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:52.473498 21182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:52.473538 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:52.473556 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:52.473573 21182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:52.473589 21182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:52.475033 21182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:52.475265 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:52.475469 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:52.480175 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:52.480217 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:52.480176 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:52.480358 21182 server_base.cc:1034] running on GCE node
I20260924 19:40:52.480667 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:52.480962 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:52.482096 21182 hybrid_clock.cc:648] HybridClock initialized: now 1790278852482081 us; error 35 us; skew 500 ppm
Sep 24 19:40:52 dist-test-slave-6kf3 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:52.489534 21182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:52.490020 21182 webserver.cc:460] Webserver started at http://127.0.0.1:38199/ using document root <none> and password file <none>
I20260924 19:40:52.490248 21182 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:52.490298 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:52.490425 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:52.491472 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "87201cd51f42458ba7700e7d9001b62e"
format_stamp: "Formatted at 2026-09-24 19:40:52 on dist-test-slave-6kf3"
server_key: "ac24e5005f99b35c75e4d6de11d97cd8"
server_key_iv: "1f9ec7fb5f1b102ce96e4ab62276b5dd"
server_key_version: "encryptionkey@0"
I20260924 19:40:52.491786 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "87201cd51f42458ba7700e7d9001b62e"
format_stamp: "Formatted at 2026-09-24 19:40:52 on dist-test-slave-6kf3"
server_key: "ac24e5005f99b35c75e4d6de11d97cd8"
server_key_iv: "1f9ec7fb5f1b102ce96e4ab62276b5dd"
server_key_version: "encryptionkey@0"
I20260924 19:40:52.493054 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:40:52.493755 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:52.493903 21182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:52.493988 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "87201cd51f42458ba7700e7d9001b62e"
format_stamp: "Formatted at 2026-09-24 19:40:52 on dist-test-slave-6kf3"
server_key: "ac24e5005f99b35c75e4d6de11d97cd8"
server_key_iv: "1f9ec7fb5f1b102ce96e4ab62276b5dd"
server_key_version: "encryptionkey@0"
I20260924 19:40:52.494052 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:52.497318 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:52.499034 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:52.499161 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:52.502919 21182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40875
I20260924 19:40:52.502954 21249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40875 every 8 connection(s)
I20260924 19:40:52.503261 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:52.504241 21250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:52.505422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21182
I20260924 19:40:52.505532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:52.505679 392 external_mini_cluster.cc:1428] Setting key 860ecf2a75b399765fcefcf43bf356f2
I20260924 19:40:52.506686 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e: Bootstrap starting.
I20260924 19:40:52.507325 21250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:52.507622 21250 log.cc:824] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:52.508016 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:52.506159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58277 (local address 127.0.0.1:40875)
0924 19:40:52.506353 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:52.506361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:52.506385 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:40:52.506793 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:52.506800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:52.506827 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:52.506940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:52.507522 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:52.507542 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:52.507542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:52.507546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:52.507583 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:52.507593 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:52.507644 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:52.507722 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:40:52.507838 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260924 19:40:52.508314 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e: No bootstrap required, opened a new log
I20260924 19:40:52.508953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:52.510335 21250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260924 19:40:52.510507 21250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:52.510552 21250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87201cd51f42458ba7700e7d9001b62e, State: Initialized, Role: FOLLOWER
I20260924 19:40:52.510671 21250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260924 19:40:52.510751 21250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:52.510782 21250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:52.510845 21250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:52.511183 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:52.510237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42849 (local address 127.0.0.1:40875)
0924 19:40:52.510274 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:52.510276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:52.510339 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:40:52.510404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:52.510404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:52.510408 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:52.510450 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:52.510993 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:52.510994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:52.510995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:52.510999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:52.511011 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:52.511015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:52.511058 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:52.511106 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:40:52.511116 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:40:52.511425 21250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260924 19:40:52.511523 21250 leader_election.cc:302] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87201cd51f42458ba7700e7d9001b62e; no voters:
I20260924 19:40:52.511672 21250 leader_election.cc:288] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:52.511711 21260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:52.511839 21260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [term 1 LEADER]: Becoming Leader. State: Replica: 87201cd51f42458ba7700e7d9001b62e, State: Running, Role: LEADER
I20260924 19:40:52.511914 21250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:52.511991 21260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260924 19:40:52.512331 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87201cd51f42458ba7700e7d9001b62e. Latest consensus state: current_term: 1 leader_uuid: "87201cd51f42458ba7700e7d9001b62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260924 19:40:52.512344 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87201cd51f42458ba7700e7d9001b62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87201cd51f42458ba7700e7d9001b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260924 19:40:52.512491 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:52.512559 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:52.512876 21269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:52.513324 21269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:52.514905 21269 catalog_manager.cc:1347] Generated new cluster ID: ec5d97f688d643c1a9ade4914f2879d7
I20260924 19:40:52.515002 21269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:52.549489 21269 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:52.549687 21269 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:52.554162 21269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87201cd51f42458ba7700e7d9001b62e: Generated new TSK 0
I20260924 19:40:52.554358 21269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:52.558055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21182
2026-09-24T19:40:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:52 dist-test-slave-6kf3 krb5kdc[21281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:52 dist-test-slave-6kf3 krb5kdc[21281](info): set up 2 sockets
Sep 24 19:40:52 dist-test-slave-6kf3 krb5kdc[21281](info): commencing operation
krb5kdc: starting...
W20260924 19:40:53.503820 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:53.612059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:53.691247 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:53.691460 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:53.691491 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:53.692655 21294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:53.692699 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:53.692718 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:53.692734 21294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:53.692747 21294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:53.694190 21294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:53.694432 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:53.694634 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:53.699354 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:53.699344 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:53.699369 21294 server_base.cc:1034] running on GCE node
W20260924 19:40:53.699344 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:53.699856 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:53.700153 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:53.701300 21294 hybrid_clock.cc:648] HybridClock initialized: now 1790278853701290 us; error 26 us; skew 500 ppm
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:53.703663 21294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:53.704137 21294 webserver.cc:460] Webserver started at http://127.0.0.1:36631/ using document root <none> and password file <none>
I20260924 19:40:53.704335 21294 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:53.704382 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:53.704511 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:53.705550 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "8e993212cd03430294f6e41df661b6a1"
format_stamp: "Formatted at 2026-09-24 19:40:53 on dist-test-slave-6kf3"
server_key: "2f101c6f8632da5a93f4513ebe7a2934"
server_key_iv: "79ab195c62aa5accfd7e4a6da8fd6486"
server_key_version: "encryptionkey@0"
I20260924 19:40:53.705873 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e993212cd03430294f6e41df661b6a1"
format_stamp: "Formatted at 2026-09-24 19:40:53 on dist-test-slave-6kf3"
server_key: "2f101c6f8632da5a93f4513ebe7a2934"
server_key_iv: "79ab195c62aa5accfd7e4a6da8fd6486"
server_key_version: "encryptionkey@0"
I20260924 19:40:53.707077 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:53.707782 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:53.707955 21294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:53.708043 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "8e993212cd03430294f6e41df661b6a1"
format_stamp: "Formatted at 2026-09-24 19:40:53 on dist-test-slave-6kf3"
server_key: "2f101c6f8632da5a93f4513ebe7a2934"
server_key_iv: "79ab195c62aa5accfd7e4a6da8fd6486"
server_key_version: "encryptionkey@0"
I20260924 19:40:53.708109 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:53.740615 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:53.742640 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:53.742776 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:53.746515 21294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40635
I20260924 19:40:53.746553 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40635 every 8 connection(s)
I20260924 19:40:53.746881 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:53.747669 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:53.749873 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1: Bootstrap starting.
I20260924 19:40:53.750380 21362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:53.750600 21362 log.cc:824] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:53.751236 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1: No bootstrap required, opened a new log
I20260924 19:40:53.752794 21362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260924 19:40:53.752925 21362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:53.752945 21362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e993212cd03430294f6e41df661b6a1, State: Initialized, Role: FOLLOWER
I20260924 19:40:53.753044 21362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260924 19:40:53.753090 21362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:53.753121 21362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:53.753150 21362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:53.753583 21362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260924 19:40:53.753659 21362 leader_election.cc:302] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e993212cd03430294f6e41df661b6a1; no voters:
I20260924 19:40:53.753775 21362 leader_election.cc:288] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:53.753824 21365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:53.753983 21362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:53.754004 21365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e993212cd03430294f6e41df661b6a1, State: Running, Role: LEADER
I20260924 19:40:53.754175 21365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260924 19:40:53.754540 21367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e993212cd03430294f6e41df661b6a1. Latest consensus state: current_term: 1 leader_uuid: "8e993212cd03430294f6e41df661b6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } } }
I20260924 19:40:53.754551 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e993212cd03430294f6e41df661b6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e993212cd03430294f6e41df661b6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } } }
I20260924 19:40:53.754671 21367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:53.754696 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:53.755046 21370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:53.755548 21370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:53.756421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21294
I20260924 19:40:53.756520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:53.756733 392 external_mini_cluster.cc:1428] Setting key 053a3645ac18f070b9de7b149450031e
I20260924 19:40:53.757009 21370 catalog_manager.cc:1347] Generated new cluster ID: 42e7a79b2e8f469db80cc25f6fbd691a
I20260924 19:40:53.757059 21370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:40:53.770674 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:53.757176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:41369 (local address 127.0.0.1:40635)
0924 19:40:53.757313 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:53.757322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:53.760121 (+ 2799us) server_negotiation.cc:407] Connection header received
0924 19:40:53.760247 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:53.760254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:53.760310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:53.760454 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:53.761122 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:53.762907 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:53.763920 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:53.764591 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:53.764697 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:53.767311 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:53.767315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:53.767315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:53.767327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:53.769111 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:53.769518 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:53.769521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:53.769521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:53.769614 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:53.769806 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:53.769807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:53.769807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:53.770107 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:53.770243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:53.770368 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:40:53.770463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260924 19:40:53.771697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:53.772967 21370 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:53.773672 21370 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:53.777045 21370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e993212cd03430294f6e41df661b6a1: Generated new TSK 0
I20260924 19:40:53.777278 21370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:53.779760 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:53.773146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:49193 (local address 127.0.0.1:40635)
0924 19:40:53.773176 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:53.773178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:53.773288 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:40:53.773331 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:53.773332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:53.773341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:53.773395 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:53.773931 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:53.774858 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:53.775779 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:53.776142 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:53.776206 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:53.777439 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:53.777440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:53.777441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:53.777447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:53.778805 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:53.779109 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:53.779110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:53.779111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:53.779164 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:53.779329 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:53.779329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:53.779330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:53.779440 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:53.779547 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:53.779616 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:40:53.779674 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:40:53.781441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21294
W20260924 19:40:53.785442 21279 connection.cc:535] client connection to 127.0.0.1:40635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40635): Connection reset by peer (error 104)
2026-09-24T19:40:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21393](info): set up 2 sockets
Sep 24 19:40:53 dist-test-slave-6kf3 krb5kdc[21393](info): commencing operation
krb5kdc: starting...
W20260924 19:40:54.723698 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:54 dist-test-slave-6kf3 krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:54.835196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:54.914517 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:54.914737 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:54.914768 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:54.915887 21406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:54.915959 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:54.915987 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:54.916013 21406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:54.916030 21406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:54.917479 21406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:54.917708 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:54.917909 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:54.922537 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:54.922549 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:54.922581 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:54.922860 21406 server_base.cc:1034] running on GCE node
I20260924 19:40:54.923055 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:54.923311 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:54.924449 21406 hybrid_clock.cc:648] HybridClock initialized: now 1790278854924434 us; error 31 us; skew 500 ppm
Sep 24 19:40:54 dist-test-slave-6kf3 krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:54.926769 21406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:54.927192 21406 webserver.cc:460] Webserver started at http://127.0.0.1:44181/ using document root <none> and password file <none>
I20260924 19:40:54.927369 21406 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:54.927407 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:54.927495 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:54.928509 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "9e1654d4e3a048ddba6a93166ba54c51"
format_stamp: "Formatted at 2026-09-24 19:40:54 on dist-test-slave-6kf3"
server_key: "c904e046553104ea1be91a0103ff25f4"
server_key_iv: "2e474e83c37b9f5dc4234d5e77a31362"
server_key_version: "encryptionkey@0"
I20260924 19:40:54.928804 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e1654d4e3a048ddba6a93166ba54c51"
format_stamp: "Formatted at 2026-09-24 19:40:54 on dist-test-slave-6kf3"
server_key: "c904e046553104ea1be91a0103ff25f4"
server_key_iv: "2e474e83c37b9f5dc4234d5e77a31362"
server_key_version: "encryptionkey@0"
I20260924 19:40:54.929939 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:54.930532 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:54.930658 21406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:54.930725 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "9e1654d4e3a048ddba6a93166ba54c51"
format_stamp: "Formatted at 2026-09-24 19:40:54 on dist-test-slave-6kf3"
server_key: "c904e046553104ea1be91a0103ff25f4"
server_key_iv: "2e474e83c37b9f5dc4234d5e77a31362"
server_key_version: "encryptionkey@0"
I20260924 19:40:54.930773 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:54.980084 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:54.981844 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:54.981953 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:54.985646 21406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33275
I20260924 19:40:54.985690 21473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33275 every 8 connection(s)
I20260924 19:40:54.986007 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:54.986815 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:54.989046 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51: Bootstrap starting.
I20260924 19:40:54.989571 21474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:54.989795 21474 log.cc:824] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:54.989982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21406
I20260924 19:40:54.990144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:54.990317 392 external_mini_cluster.cc:1428] Setting key e32eca6c7f1b2ec031c3302b29d50fde
I20260924 19:40:54.990538 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51: No bootstrap required, opened a new log
I20260924 19:40:54.992424 21474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } }
I20260924 19:40:54.992561 21474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:54.992594 21474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1654d4e3a048ddba6a93166ba54c51, State: Initialized, Role: FOLLOWER
I20260924 19:40:54.992727 21474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } }
I20260924 19:40:54.992799 21474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:54.992827 21474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:54.992873 21474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:54.993399 21474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } }
I20260924 19:40:54.993497 21474 leader_election.cc:302] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e1654d4e3a048ddba6a93166ba54c51; no voters:
I20260924 19:40:54.993661 21474 leader_election.cc:288] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:54.993700 21479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:54.993846 21479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [term 1 LEADER]: Becoming Leader. State: Replica: 9e1654d4e3a048ddba6a93166ba54c51, State: Running, Role: LEADER
I20260924 19:40:54.993906 21474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:54.993932 21479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } }
I20260924 19:40:54.994267 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e1654d4e3a048ddba6a93166ba54c51. Latest consensus state: current_term: 1 leader_uuid: "9e1654d4e3a048ddba6a93166ba54c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } } }
I20260924 19:40:54.994478 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:54.994414 21480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e1654d4e3a048ddba6a93166ba54c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1654d4e3a048ddba6a93166ba54c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33275 } } }
I20260924 19:40:54.994702 21480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:54.995007 21488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:54.995570 21488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:54.996681 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:54.990777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:47149 (local address 127.0.0.1:33275)
0924 19:40:54.990956 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:54.990964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:54.990988 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:40:54.991720 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:54.991727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:54.991779 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:54.991904 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:54.992142 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:54.993370 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:54.994383 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:54.994964 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:54.995055 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:54.996006 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:54.996010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:54.996011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:54.996205 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:54.996242 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:54.996257 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:54.996337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:54.996433 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:40:54.996460 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260924 19:40:54.997187 21488 catalog_manager.cc:1347] Generated new cluster ID: d72bcd2a58e4425881a6d11b89576d67
I20260924 19:40:54.997231 21488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:54.997594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:55.002331 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:54.999051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:43781 (local address 127.0.0.1:33275)
0924 19:40:54.999086 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:54.999088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:54.999263 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:40:54.999293 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:54.999293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:54.999303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:54.999363 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:54.999486 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:55.000210 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:55.001173 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:55.001474 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:55.001550 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:55.002103 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:55.002104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:55.002105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:55.002109 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:55.002123 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:55.002127 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:55.002181 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:55.002247 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:40:55.002257 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:40:55.022697 21488 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:55.023437 21488 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:55.039793 21488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e1654d4e3a048ddba6a93166ba54c51: Generated new TSK 0
I20260924 19:40:55.040046 21488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:55.049288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21406
W20260924 19:40:55.053061 21391 connection.cc:535] client connection to 127.0.0.1:33275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33275): Connection reset by peer (error 104)
2026-09-24T19:40:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:55 dist-test-slave-6kf3 krb5kdc[21505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:55 dist-test-slave-6kf3 krb5kdc[21505](info): set up 2 sockets
Sep 24 19:40:55 dist-test-slave-6kf3 krb5kdc[21505](info): commencing operation
krb5kdc: starting...
W20260924 19:40:56.000972 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:56 dist-test-slave-6kf3 krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:56.112985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:56.190832 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:56.191031 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:56.191052 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:56.192232 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:56.192273 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:56.192286 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:56.192305 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:56.192330 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:56.193768 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:56.193991 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:56.194197 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:40:56.198836 21518 server_base.cc:1034] running on GCE node
W20260924 19:40:56.198820 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:56.198814 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:56.198830 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:56.199242 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:56.199510 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:56.200641 21518 hybrid_clock.cc:648] HybridClock initialized: now 1790278856200625 us; error 31 us; skew 500 ppm
Sep 24 19:40:56 dist-test-slave-6kf3 krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:56.202926 21518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:56.203349 21518 webserver.cc:460] Webserver started at http://127.0.0.1:39199/ using document root <none> and password file <none>
I20260924 19:40:56.203544 21518 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:56.203590 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:56.203724 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:56.204795 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "2dae49413c6d426e87cd6e324a1c6cae"
format_stamp: "Formatted at 2026-09-24 19:40:56 on dist-test-slave-6kf3"
server_key: "107f3d7eb5af54b3d8be801db6c125bd"
server_key_iv: "d5540165d8e90431a2e6d8e8539c4e98"
server_key_version: "encryptionkey@0"
I20260924 19:40:56.205118 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dae49413c6d426e87cd6e324a1c6cae"
format_stamp: "Formatted at 2026-09-24 19:40:56 on dist-test-slave-6kf3"
server_key: "107f3d7eb5af54b3d8be801db6c125bd"
server_key_iv: "d5540165d8e90431a2e6d8e8539c4e98"
server_key_version: "encryptionkey@0"
I20260924 19:40:56.206284 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:56.206954 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:56.207111 21518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:56.207192 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "2dae49413c6d426e87cd6e324a1c6cae"
format_stamp: "Formatted at 2026-09-24 19:40:56 on dist-test-slave-6kf3"
server_key: "107f3d7eb5af54b3d8be801db6c125bd"
server_key_iv: "d5540165d8e90431a2e6d8e8539c4e98"
server_key_version: "encryptionkey@0"
I20260924 19:40:56.207258 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:56.210450 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:56.212201 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:56.212318 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:56.216123 21518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45795
I20260924 19:40:56.216149 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45795 every 8 connection(s)
I20260924 19:40:56.216434 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:56.217139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21518
I20260924 19:40:56.217234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:56.217346 21586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:56.217409 392 external_mini_cluster.cc:1428] Setting key 3a5517549f857e99f294aa379ceb0f97
I20260924 19:40:56.219678 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:56.217862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:35549 (local address 127.0.0.1:45795)
0924 19:40:56.218044 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:56.218052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:56.218079 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:40:56.218550 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:56.218556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:56.218581 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:56.218715 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:56.219314 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:56.219318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:56.219318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:56.219323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:56.219360 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:56.219370 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:56.219422 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:56.219489 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:40:56.219528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260924 19:40:56.220196 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: Bootstrap starting.
I20260924 19:40:56.220597 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:56.220829 21586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:56.221076 21586 log.cc:824] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:56.221711 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: No bootstrap required, opened a new log
I20260924 19:40:56.222782 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:56.221841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:55877 (local address 127.0.0.1:45795)
0924 19:40:56.221886 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:56.221887 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:56.222013 (+ 126us) server_negotiation.cc:407] Connection header received
0924 19:40:56.222076 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:56.222076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:56.222081 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:56.222137 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:56.222605 (+ 468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:56.222606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:56.222606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:56.222609 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:56.222620 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:56.222624 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:56.222666 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:56.222713 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:40:56.222724 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:40:56.223845 21586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260924 19:40:56.224012 21586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:56.224058 21586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dae49413c6d426e87cd6e324a1c6cae, State: Initialized, Role: FOLLOWER
I20260924 19:40:56.224174 21586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260924 19:40:56.224253 21586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:56.224285 21586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:56.224331 21586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:56.224812 21586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260924 19:40:56.224895 21586 leader_election.cc:302] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dae49413c6d426e87cd6e324a1c6cae; no voters:
I20260924 19:40:56.225056 21586 leader_election.cc:288] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:56.225106 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:56.225263 21586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:56.225289 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [term 1 LEADER]: Becoming Leader. State: Replica: 2dae49413c6d426e87cd6e324a1c6cae, State: Running, Role: LEADER
I20260924 19:40:56.225376 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260924 19:40:56.225709 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dae49413c6d426e87cd6e324a1c6cae. Latest consensus state: current_term: 1 leader_uuid: "2dae49413c6d426e87cd6e324a1c6cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } } }
I20260924 19:40:56.225800 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:56.225978 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dae49413c6d426e87cd6e324a1c6cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dae49413c6d426e87cd6e324a1c6cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } } }
I20260924 19:40:56.226086 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae [sys.catalog]: This master's current role is: LEADER
W20260924 19:40:56.226814 21612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:40:56.226855 21612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:40:56.226904 21605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:56.227057 21605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:56.228305 21605 catalog_manager.cc:1347] Generated new cluster ID: 9af43ab0050c4fecba064bef2a9c0974
I20260924 19:40:56.228355 21605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:56.238243 21605 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:56.238396 21605 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:56.242797 21605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dae49413c6d426e87cd6e324a1c6cae: Generated new TSK 0
I20260924 19:40:56.242980 21605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:56.247474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21518
2026-09-24T19:40:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:56 dist-test-slave-6kf3 krb5kdc[21617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:56 dist-test-slave-6kf3 krb5kdc[21617](info): set up 2 sockets
Sep 24 19:40:56 dist-test-slave-6kf3 krb5kdc[21617](info): commencing operation
krb5kdc: starting...
W20260924 19:40:57.195796 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:57.306128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:57.383662 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:57.383864 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:57.383893 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:57.385039 21630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:57.385080 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:57.385099 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:57.385115 21630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:57.385131 21630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:57.386569 21630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:57.386804 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:57.387001 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:57.391537 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:57.391582 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:57.391561 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:57.391798 21630 server_base.cc:1034] running on GCE node
I20260924 19:40:57.392047 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:57.392295 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:57.393421 21630 hybrid_clock.cc:648] HybridClock initialized: now 1790278857393411 us; error 26 us; skew 500 ppm
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:57.395697 21630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:57.396117 21630 webserver.cc:460] Webserver started at http://127.0.0.1:38785/ using document root <none> and password file <none>
I20260924 19:40:57.396334 21630 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:57.396378 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:57.396474 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:57.397461 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "4525ca769e6749a2afe71b6aa5350030"
format_stamp: "Formatted at 2026-09-24 19:40:57 on dist-test-slave-6kf3"
server_key: "85a0238b09e8ece8bb4c23a52f4b909e"
server_key_iv: "bccda435de62a0bba31c190f82a9cc92"
server_key_version: "encryptionkey@0"
I20260924 19:40:57.397746 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "4525ca769e6749a2afe71b6aa5350030"
format_stamp: "Formatted at 2026-09-24 19:40:57 on dist-test-slave-6kf3"
server_key: "85a0238b09e8ece8bb4c23a52f4b909e"
server_key_iv: "bccda435de62a0bba31c190f82a9cc92"
server_key_version: "encryptionkey@0"
I20260924 19:40:57.398873 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:40:57.399446 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:57.399609 21630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:57.399683 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "4525ca769e6749a2afe71b6aa5350030"
format_stamp: "Formatted at 2026-09-24 19:40:57 on dist-test-slave-6kf3"
server_key: "85a0238b09e8ece8bb4c23a52f4b909e"
server_key_iv: "bccda435de62a0bba31c190f82a9cc92"
server_key_version: "encryptionkey@0"
I20260924 19:40:57.399744 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:57.409901 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:57.411852 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:57.411993 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:57.415472 21630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33295
I20260924 19:40:57.415544 21697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33295 every 8 connection(s)
I20260924 19:40:57.415814 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:57.416677 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:57.419000 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030: Bootstrap starting.
I20260924 19:40:57.419687 21698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:57.419919 21698 log.cc:824] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:57.420269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21630
I20260924 19:40:57.420364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:57.420544 392 external_mini_cluster.cc:1428] Setting key af8a09a123c2c6c29166098f0561bab4
I20260924 19:40:57.420701 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030: No bootstrap required, opened a new log
I20260924 19:40:57.422827 21698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } }
I20260924 19:40:57.422976 21698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:57.423008 21698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4525ca769e6749a2afe71b6aa5350030, State: Initialized, Role: FOLLOWER
I20260924 19:40:57.423126 21698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } }
I20260924 19:40:57.423197 21698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:57.423224 21698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:57.423269 21698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:57.423772 21698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } }
I20260924 19:40:57.423861 21698 leader_election.cc:302] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4525ca769e6749a2afe71b6aa5350030; no voters:
I20260924 19:40:57.424036 21698 leader_election.cc:288] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:57.424070 21703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:57.424170 21703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [term 1 LEADER]: Becoming Leader. State: Replica: 4525ca769e6749a2afe71b6aa5350030, State: Running, Role: LEADER
I20260924 19:40:57.424278 21698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:57.424305 21703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } }
I20260924 19:40:57.424655 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4525ca769e6749a2afe71b6aa5350030. Latest consensus state: current_term: 1 leader_uuid: "4525ca769e6749a2afe71b6aa5350030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } } }
I20260924 19:40:57.424793 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:57.425037 21712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:57.425707 21712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:57.426627 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4525ca769e6749a2afe71b6aa5350030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525ca769e6749a2afe71b6aa5350030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33295 } } }
I20260924 19:40:57.426771 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:57.427274 21712 catalog_manager.cc:1347] Generated new cluster ID: 69de0668caeb4981b2188b5beee0c8f2
I20260924 19:40:57.427322 21712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:40:57.432520 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:57.421096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:33961 (local address 127.0.0.1:33295)
0924 19:40:57.421261 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:57.421271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:57.421297 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:40:57.421899 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:57.421906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:57.421960 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:57.422075 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:57.422836 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:57.424159 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:57.425252 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:57.425850 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:57.425970 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:57.428879 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:57.428883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:57.428884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:57.428895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:57.430705 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:57.431157 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:57.431159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:57.431159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:57.431253 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:57.431483 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:57.431484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:57.431484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:57.431754 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:57.431890 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:57.432089 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:40:57.432184 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260924 19:40:57.433490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:57.438413 21712 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:57.439182 21712 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:57.441926 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:57.434938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:60033 (local address 127.0.0.1:33295)
0924 19:40:57.434970 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:57.434972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:57.435092 (+ 120us) server_negotiation.cc:407] Connection header received
0924 19:40:57.435160 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:57.435160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:57.435169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:57.435218 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:57.435873 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:57.436633 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:57.437606 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:57.437905 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:57.437965 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:57.439356 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:57.439357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:57.439358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:40:57.439364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:57.440834 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:57.441194 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:57.441195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:57.441196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:57.441240 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:40:57.441421 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:40:57.441421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:40:57.441421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:40:57.441531 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:57.441596 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:57.441788 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:40:57.441842 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:40:57.445864 21712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4525ca769e6749a2afe71b6aa5350030: Generated new TSK 0
I20260924 19:40:57.446028 21712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:57.451447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21630
W20260924 19:40:57.455089 21615 connection.cc:535] client connection to 127.0.0.1:33295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33295): Connection reset by peer (error 104)
2026-09-24T19:40:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21729](info): set up 2 sockets
Sep 24 19:40:57 dist-test-slave-6kf3 krb5kdc[21729](info): commencing operation
krb5kdc: starting...
W20260924 19:40:58.399991 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:58 dist-test-slave-6kf3 krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:58.510795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:58.588589 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:58.588788 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:58.588819 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:58.589936 21742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:58.589977 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:58.589993 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:58.590009 21742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:58.590023 21742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:58.591464 21742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:58.591699 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:58.591890 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:58.596349 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:58.596352 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:58.596406 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:58.596431 21742 server_base.cc:1034] running on GCE node
I20260924 19:40:58.596786 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:58.597044 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:58.598181 21742 hybrid_clock.cc:648] HybridClock initialized: now 1790278858598170 us; error 28 us; skew 500 ppm
Sep 24 19:40:58 dist-test-slave-6kf3 krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:58.600495 21742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:58.600878 21742 webserver.cc:460] Webserver started at http://127.0.0.1:36407/ using document root <none> and password file <none>
I20260924 19:40:58.601080 21742 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:58.601145 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:58.601266 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:58.602250 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "be25157eeaf947cdafc4fabf7e6af9bc"
format_stamp: "Formatted at 2026-09-24 19:40:58 on dist-test-slave-6kf3"
server_key: "919364f04a6ee3de625b6e8880f4720e"
server_key_iv: "68338ced3f96e1cfb0c9ccc58ab6d6d7"
server_key_version: "encryptionkey@0"
I20260924 19:40:58.602555 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "be25157eeaf947cdafc4fabf7e6af9bc"
format_stamp: "Formatted at 2026-09-24 19:40:58 on dist-test-slave-6kf3"
server_key: "919364f04a6ee3de625b6e8880f4720e"
server_key_iv: "68338ced3f96e1cfb0c9ccc58ab6d6d7"
server_key_version: "encryptionkey@0"
I20260924 19:40:58.603715 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:40:58.604396 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:58.604534 21742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:58.604610 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "be25157eeaf947cdafc4fabf7e6af9bc"
format_stamp: "Formatted at 2026-09-24 19:40:58 on dist-test-slave-6kf3"
server_key: "919364f04a6ee3de625b6e8880f4720e"
server_key_iv: "68338ced3f96e1cfb0c9ccc58ab6d6d7"
server_key_version: "encryptionkey@0"
I20260924 19:40:58.604671 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:58.616202 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:58.618191 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:58.618314 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:58.621906 21742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44249
I20260924 19:40:58.621948 21809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44249 every 8 connection(s)
I20260924 19:40:58.622254 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:58.623208 21810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:58.625017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21742
I20260924 19:40:58.625142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:58.625319 392 external_mini_cluster.cc:1428] Setting key bbb94eda6044c9f4487144a2aade5824
I20260924 19:40:58.625741 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc: Bootstrap starting.
I20260924 19:40:58.626354 21810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:58.626623 21810 log.cc:824] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:58.627270 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc: No bootstrap required, opened a new log
I20260924 19:40:58.629325 21810 raft_consensus.cc:348] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260924 19:40:58.629492 21810 raft_consensus.cc:374] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:58.629539 21810 raft_consensus.cc:729] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25157eeaf947cdafc4fabf7e6af9bc, State: Initialized, Role: FOLLOWER
I20260924 19:40:58.629671 21810 consensus_queue.cc:260] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260924 19:40:58.629748 21810 raft_consensus.cc:388] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:58.629770 21810 raft_consensus.cc:482] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:58.629814 21810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:58.630352 21810 raft_consensus.cc:504] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260924 19:40:58.630458 21810 leader_election.cc:302] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be25157eeaf947cdafc4fabf7e6af9bc; no voters:
I20260924 19:40:58.630621 21810 leader_election.cc:288] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:58.630640 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:58.625805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:33149 (local address 127.0.0.1:44249)
0924 19:40:58.625963 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:58.625972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:58.625996 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:40:58.626415 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:58.626422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:58.626485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:58.626616 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:58.626820 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:58.627824 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:58.628937 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:58.629322 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:58.629397 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:58.630037 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:58.630041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:58.630041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:58.630196 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:58.630225 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:58.630235 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:58.630293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:58.630405 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:40:58.630429 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260924 19:40:58.630649 21815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:58.630771 21815 raft_consensus.cc:686] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [term 1 LEADER]: Becoming Leader. State: Replica: be25157eeaf947cdafc4fabf7e6af9bc, State: Running, Role: LEADER
I20260924 19:40:58.630859 21810 sys_catalog.cc:564] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:58.630867 21815 consensus_queue.cc:237] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } }
I20260924 19:40:58.631362 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } } }
I20260924 19:40:58.631489 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:58.631552 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:58.631644 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader be25157eeaf947cdafc4fabf7e6af9bc. Latest consensus state: current_term: 1 leader_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be25157eeaf947cdafc4fabf7e6af9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44249 } } }
I20260924 19:40:58.631690 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:58.632117 21824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:58.633383 21824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:58.634707 21824 catalog_manager.cc:1347] Generated new cluster ID: 402c6d96e4904024a7f7d95de8065911
I20260924 19:40:58.634757 21824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:58.641358 21824 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:58.642038 21824 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:58.642339 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:58.632872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:56175 (local address 127.0.0.1:44249)
0924 19:40:58.632914 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:58.632916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:58.633080 (+ 164us) server_negotiation.cc:407] Connection header received
0924 19:40:58.633129 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:58.633130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:58.633138 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:58.633176 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:58.633304 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:58.633977 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:58.641136 (+ 7159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:40:58.641413 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:40:58.641460 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:40:58.642124 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:58.642125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:58.642125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:58.642130 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:58.642144 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:58.642148 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:58.642193 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:58.642260 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:40:58.642269 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:40:58.649448 21824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be25157eeaf947cdafc4fabf7e6af9bc: Generated new TSK 0
I20260924 19:40:58.649631 21824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:58.656687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21742
W20260924 19:40:58.660431 21727 connection.cc:535] client connection to 127.0.0.1:44249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44249): Connection reset by peer (error 104)
2026-09-24T19:40:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:58 dist-test-slave-6kf3 krb5kdc[21841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:58 dist-test-slave-6kf3 krb5kdc[21841](info): set up 2 sockets
Sep 24 19:40:58 dist-test-slave-6kf3 krb5kdc[21841](info): commencing operation
krb5kdc: starting...
W20260924 19:40:59.603803 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:40:59 dist-test-slave-6kf3 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:40:59.715363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:40:59.794340 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:40:59.794550 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:40:59.794581 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:40:59.795698 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:40:59.795739 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:40:59.795758 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:40:59.795773 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:40:59.795789 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:40:59.797256 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:40:59.797497 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:40:59.797698 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:40:59.802289 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:40:59.802295 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:59.802416 21854 server_base.cc:1034] running on GCE node
W20260924 19:40:59.802304 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:40:59.802686 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:40:59.802951 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:40:59.804075 21854 hybrid_clock.cc:648] HybridClock initialized: now 1790278859804064 us; error 27 us; skew 500 ppm
Sep 24 19:40:59 dist-test-slave-6kf3 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:40:59.806267 21854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:40:59.806668 21854 webserver.cc:460] Webserver started at http://127.0.0.1:35645/ using document root <none> and password file <none>
I20260924 19:40:59.806859 21854 fs_manager.cc:359] Metadata directory not provided
I20260924 19:40:59.806903 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:40:59.807022 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:40:59.808055 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "06c3b621acbc471d9c96a06759dc31b8"
format_stamp: "Formatted at 2026-09-24 19:40:59 on dist-test-slave-6kf3"
server_key: "e8234bc1eb1e9dee5022bf6c37feac0a"
server_key_iv: "355af1a151f237b66fb8270747de700e"
server_key_version: "encryptionkey@0"
I20260924 19:40:59.808372 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "06c3b621acbc471d9c96a06759dc31b8"
format_stamp: "Formatted at 2026-09-24 19:40:59 on dist-test-slave-6kf3"
server_key: "e8234bc1eb1e9dee5022bf6c37feac0a"
server_key_iv: "355af1a151f237b66fb8270747de700e"
server_key_version: "encryptionkey@0"
I20260924 19:40:59.809536 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:40:59.810199 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:40:59.810346 21854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:40:59.810424 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "06c3b621acbc471d9c96a06759dc31b8"
format_stamp: "Formatted at 2026-09-24 19:40:59 on dist-test-slave-6kf3"
server_key: "e8234bc1eb1e9dee5022bf6c37feac0a"
server_key_iv: "355af1a151f237b66fb8270747de700e"
server_key_version: "encryptionkey@0"
I20260924 19:40:59.810487 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:40:59.813592 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:40:59.815209 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:40:59.815332 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:40:59.819085 21854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45179
I20260924 19:40:59.819118 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45179 every 8 connection(s)
I20260924 19:40:59.819485 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:40:59.820495 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:40:59.822620 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8: Bootstrap starting.
I20260924 19:40:59.823167 21922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8: Neither blocks nor log segments found. Creating new log.
I20260924 19:40:59.823415 21922 log.cc:824] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:40:59.824043 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8: No bootstrap required, opened a new log
I20260924 19:40:59.825560 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } }
I20260924 19:40:59.825703 21922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:40:59.825734 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c3b621acbc471d9c96a06759dc31b8, State: Initialized, Role: FOLLOWER
I20260924 19:40:59.825870 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } }
I20260924 19:40:59.825945 21922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:40:59.825984 21922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:40:59.826030 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:40:59.826515 21922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } }
I20260924 19:40:59.826604 21922 leader_election.cc:302] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c3b621acbc471d9c96a06759dc31b8; no voters:
I20260924 19:40:59.826767 21922 leader_election.cc:288] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:40:59.826810 21925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:40:59.826993 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:40:59.827144 21925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [term 1 LEADER]: Becoming Leader. State: Replica: 06c3b621acbc471d9c96a06759dc31b8, State: Running, Role: LEADER
I20260924 19:40:59.827272 21925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } }
I20260924 19:40:59.827612 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c3b621acbc471d9c96a06759dc31b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } } }
I20260924 19:40:59.827623 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c3b621acbc471d9c96a06759dc31b8. Latest consensus state: current_term: 1 leader_uuid: "06c3b621acbc471d9c96a06759dc31b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3b621acbc471d9c96a06759dc31b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45179 } } }
I20260924 19:40:59.827703 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:59.827710 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:40:59.827896 21934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:40:59.828442 21934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:40:59.829604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21854
I20260924 19:40:59.829730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:40:59.829840 21934 catalog_manager.cc:1347] Generated new cluster ID: 6f7b5fb20ab4464aa9bda840afd9ccd6
I20260924 19:40:59.829890 21934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:40:59.829907 392 external_mini_cluster.cc:1428] Setting key c20961ebc134b7c47a0895461dd48620
I20260924 19:40:59.836223 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:59.834483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:54739 (local address 127.0.0.1:45179)
0924 19:40:59.834728 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:59.834738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:59.834773 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:40:59.834897 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:59.834903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:59.834941 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:59.835056 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:59.835759 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:59.835762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:59.835763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:59.835767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:59.835804 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:59.835815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:59.835865 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:59.835945 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:40:59.836086 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
I20260924 19:40:59.837047 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:40:59.841676 21934 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:40:59.841840 21934 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:40:59.844738 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:40:59.841554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:60933 (local address 127.0.0.1:45179)
0924 19:40:59.841687 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 19:40:59.841689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:40:59.841698 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:40:59.843850 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:40:59.843851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:40:59.843857 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:40:59.843901 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:40:59.844520 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:40:59.844521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:40:59.844521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:40:59.844525 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:40:59.844541 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:40:59.844546 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:40:59.844593 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:40:59.844641 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:40:59.844656 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 19:40:59.846302 21934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c3b621acbc471d9c96a06759dc31b8: Generated new TSK 0
I20260924 19:40:59.846498 21934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:40:59.848742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21854
2026-09-24T19:40:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1191 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:40:59 dist-test-slave-6kf3 krb5kdc[21953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:40:59 dist-test-slave-6kf3 krb5kdc[21953](info): set up 2 sockets
Sep 24 19:40:59 dist-test-slave-6kf3 krb5kdc[21953](info): commencing operation
krb5kdc: starting...
W20260924 19:41:00.815801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:00 dist-test-slave-6kf3 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:00.930557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:01.009385 21966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:01.009590 21966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:01.009613 21966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:01.010752 21966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:01.010794 21966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:01.010807 21966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:01.010818 21966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:01.010831 21966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:01.012323 21966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:01.012557 21966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:01.012781 21966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:41:01.017426 21966 server_base.cc:1034] running on GCE node
W20260924 19:41:01.017381 21971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:01.017416 21972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:01.017387 21974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:01.017771 21966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:01.018049 21966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:01.019196 21966 hybrid_clock.cc:648] HybridClock initialized: now 1790278861019180 us; error 27 us; skew 500 ppm
Sep 24 19:41:01 dist-test-slave-6kf3 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:01.021601 21966 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:01.022029 21966 webserver.cc:460] Webserver started at http://127.0.98.62:40285/ using document root <none> and password file <none>
I20260924 19:41:01.022209 21966 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:01.022248 21966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:01.022336 21966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:01.023324 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "25a940f16e0a434fbfc97917a6ea0810"
format_stamp: "Formatted at 2026-09-24 19:41:01 on dist-test-slave-6kf3"
server_key: "b780a75a065fa7fa7a00fa40ddc6c459"
server_key_iv: "30cbd195e38507e724bc9ef7fef4d371"
server_key_version: "encryptionkey@0"
I20260924 19:41:01.023612 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "25a940f16e0a434fbfc97917a6ea0810"
format_stamp: "Formatted at 2026-09-24 19:41:01 on dist-test-slave-6kf3"
server_key: "b780a75a065fa7fa7a00fa40ddc6c459"
server_key_iv: "30cbd195e38507e724bc9ef7fef4d371"
server_key_version: "encryptionkey@0"
I20260924 19:41:01.024854 21966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:41:01.025477 21981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:01.025624 21966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:41:01.025707 21966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "25a940f16e0a434fbfc97917a6ea0810"
format_stamp: "Formatted at 2026-09-24 19:41:01 on dist-test-slave-6kf3"
server_key: "b780a75a065fa7fa7a00fa40ddc6c459"
server_key_iv: "30cbd195e38507e724bc9ef7fef4d371"
server_key_version: "encryptionkey@0"
I20260924 19:41:01.025763 21966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:01.042201 21966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:01.044317 21966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:01.044430 21966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:01.048089 21966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37429
I20260924 19:41:01.048153 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37429 every 8 connection(s)
I20260924 19:41:01.048455 21966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:01.049297 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:01.051683 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810: Bootstrap starting.
I20260924 19:41:01.052276 22034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:01.052534 22034 log.cc:824] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:01.053189 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810: No bootstrap required, opened a new log
I20260924 19:41:01.054965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 21966
I20260924 19:41:01.054876 22034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260924 19:41:01.055012 22034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:01.055043 22034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a940f16e0a434fbfc97917a6ea0810, State: Initialized, Role: FOLLOWER
I20260924 19:41:01.055059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:01.055217 392 external_mini_cluster.cc:1428] Setting key 9daa8d702c758dd0502ad06af7ecee73
I20260924 19:41:01.055189 22034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260924 19:41:01.055262 22034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:01.055300 22034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:01.055347 22034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:01.056030 22034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260924 19:41:01.056128 22034 leader_election.cc:302] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a940f16e0a434fbfc97917a6ea0810; no voters:
I20260924 19:41:01.056303 22034 leader_election.cc:288] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:01.056337 22039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:01.056433 22039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [term 1 LEADER]: Becoming Leader. State: Replica: 25a940f16e0a434fbfc97917a6ea0810, State: Running, Role: LEADER
I20260924 19:41:01.056571 22034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:01.056568 22039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260924 19:41:01.057343 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25a940f16e0a434fbfc97917a6ea0810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } } }
I20260924 19:41:01.057449 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:01.057668 22048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:01.058382 22048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:01.058171 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a940f16e0a434fbfc97917a6ea0810. Latest consensus state: current_term: 1 leader_uuid: "25a940f16e0a434fbfc97917a6ea0810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a940f16e0a434fbfc97917a6ea0810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } } }
I20260924 19:41:01.059157 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:01.060695 22048 catalog_manager.cc:1347] Generated new cluster ID: 8f3af34fa24b43d5b8e8b06b0f5a4498
I20260924 19:41:01.060758 22048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:41:01 dist-test-slave-6kf3 krb5kdc[21953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:01.065842 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:01.055671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38975 (local address 127.0.98.62:37429)
0924 19:41:01.056134 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:01.056145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:01.056179 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:41:01.056288 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:01.056294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:01.056339 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:01.057259 (+ 920us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:01.057355 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:01.058915 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:01.059787 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:01.060406 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:01.060528 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:01.062794 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:01.062798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:01.062799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:01.062812 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:01.064339 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:01.064710 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:01.064712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:01.064713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:01.064844 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:01.065050 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:01.065051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:01.065051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:01.065282 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:01.065436 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:01.065547 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:41:01.065656 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":375,"threads_started":1}
I20260924 19:41:01.066682 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:01.067505 22048 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:01.068217 22048 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:01.073630 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:01.068017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45835 (local address 127.0.98.62:37429)
0924 19:41:01.068048 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:01.068050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:01.068152 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:41:01.068207 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:01.068207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:01.068216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:01.068277 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:01.068770 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:01.069287 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:01.069947 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:01.070168 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:01.070219 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:01.071265 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:01.071266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:01.071267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:01.071273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:01.072536 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:01.072871 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:01.072872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:01.072872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:01.072924 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:01.073102 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:01.073103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:01.073110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:01.073208 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:01.073294 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:01.073407 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 19:41:01.073462 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":16}
I20260924 19:41:01.078789 22048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25a940f16e0a434fbfc97917a6ea0810: Generated new TSK 0
I20260924 19:41:01.078984 22048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:01.081547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 21966
W20260924 19:41:01.085485 21951 connection.cc:535] client connection to 127.0.98.62:37429 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37429): Connection reset by peer (error 104)
2026-09-24T19:41:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:01 dist-test-slave-6kf3 krb5kdc[22065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:01 dist-test-slave-6kf3 krb5kdc[22065](info): set up 2 sockets
Sep 24 19:41:01 dist-test-slave-6kf3 krb5kdc[22065](info): commencing operation
krb5kdc: starting...
W20260924 19:41:02.023726 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:02 dist-test-slave-6kf3 krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:02.136320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:02.214865 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:02.215071 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:02.215102 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:02.216287 22078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:02.216328 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:02.216347 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:02.216362 22078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:02.216380 22078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:02.217819 22078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:02.218071 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:02.218271 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:02.222782 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:02.222868 22078 server_base.cc:1034] running on GCE node
W20260924 19:41:02.222788 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:02.222788 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:02.223228 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:02.223517 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:02.224651 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790278862224640 us; error 27 us; skew 500 ppm
Sep 24 19:41:02 dist-test-slave-6kf3 krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:02.226902 22078 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:02.227276 22078 webserver.cc:460] Webserver started at http://127.0.98.62:40489/ using document root <none> and password file <none>
I20260924 19:41:02.227463 22078 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:02.227507 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:02.227632 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:02.228675 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "249643c708ac404d8f963b1dc468d98c"
format_stamp: "Formatted at 2026-09-24 19:41:02 on dist-test-slave-6kf3"
server_key: "f58703ee338a515170ddf31cd85f62d9"
server_key_iv: "52f7ef4098d21d6ec2a0a389cf969281"
server_key_version: "encryptionkey@0"
I20260924 19:41:02.228989 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "249643c708ac404d8f963b1dc468d98c"
format_stamp: "Formatted at 2026-09-24 19:41:02 on dist-test-slave-6kf3"
server_key: "f58703ee338a515170ddf31cd85f62d9"
server_key_iv: "52f7ef4098d21d6ec2a0a389cf969281"
server_key_version: "encryptionkey@0"
I20260924 19:41:02.230201 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:41:02.230868 22093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:02.231029 22078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:41:02.231109 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "249643c708ac404d8f963b1dc468d98c"
format_stamp: "Formatted at 2026-09-24 19:41:02 on dist-test-slave-6kf3"
server_key: "f58703ee338a515170ddf31cd85f62d9"
server_key_iv: "52f7ef4098d21d6ec2a0a389cf969281"
server_key_version: "encryptionkey@0"
I20260924 19:41:02.231168 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:02.241534 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:02.243539 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:02.243676 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:02.247267 22078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34491
I20260924 19:41:02.247623 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34491 every 8 connection(s)
I20260924 19:41:02.247711 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:02.248373 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:02.250415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22078
I20260924 19:41:02.250517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:02.250690 392 external_mini_cluster.cc:1428] Setting key dfad29c419a07b7b5af7d936f27548f3
I20260924 19:41:02.250737 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c: Bootstrap starting.
I20260924 19:41:02.251421 22146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:02.251732 22146 log.cc:824] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:02.252487 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c: No bootstrap required, opened a new log
I20260924 19:41:02.254709 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260924 19:41:02.254874 22146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:02.254905 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249643c708ac404d8f963b1dc468d98c, State: Initialized, Role: FOLLOWER
I20260924 19:41:02.255025 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260924 19:41:02.255101 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:02.255133 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:02.255177 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:02.255702 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260924 19:41:02.255803 22146 leader_election.cc:302] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249643c708ac404d8f963b1dc468d98c; no voters:
I20260924 19:41:02.255980 22146 leader_election.cc:288] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:02.256014 22151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:02.256160 22151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [term 1 LEADER]: Becoming Leader. State: Replica: 249643c708ac404d8f963b1dc468d98c, State: Running, Role: LEADER
I20260924 19:41:02.256230 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:02.256254 22151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260924 19:41:02.256551 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249643c708ac404d8f963b1dc468d98c. Latest consensus state: current_term: 1 leader_uuid: "249643c708ac404d8f963b1dc468d98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } } }
I20260924 19:41:02.256632 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:02.256640 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:02.251175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54653 (local address 127.0.98.62:34491)
0924 19:41:02.251371 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:02.251381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:02.251416 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:41:02.252010 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:02.252019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:02.252076 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:02.252201 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:02.252381 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:02.253706 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:02.254465 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:02.255061 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:02.255166 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:02.255803 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:02.255807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:02.255808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:02.256029 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:02.256080 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:02.256096 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:02.256178 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:02.256315 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:41:02.256449 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260924 19:41:02.256559 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249643c708ac404d8f963b1dc468d98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249643c708ac404d8f963b1dc468d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } } }
I20260924 19:41:02.256896 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:02.257475 22160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:02.257768 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:02.258113 22160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:02.260315 22160 catalog_manager.cc:1347] Generated new cluster ID: 73ffb0ae985c4f548e66400f3a4ea738
I20260924 19:41:02.260365 22160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:02.262280 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:02.259213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36153 (local address 127.0.98.62:34491)
0924 19:41:02.259242 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:02.259243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:02.259354 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:41:02.259407 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:02.259407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:02.259416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:02.259460 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:02.259575 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:02.260098 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:02.261081 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:02.261397 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:02.261471 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:02.262037 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:02.262038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:02.262039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:02.262044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:02.262057 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:02.262062 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:02.262123 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:02.262196 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:41:02.262206 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 19:41:02.271308 22160 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:02.271884 22160 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:02.274273 22160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249643c708ac404d8f963b1dc468d98c: Generated new TSK 0
I20260924 19:41:02.274423 22160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:02.277869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22078
W20260924 19:41:02.281596 22063 connection.cc:535] client connection to 127.0.98.62:34491 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34491): Connection reset by peer (error 104)
2026-09-24T19:41:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:02 dist-test-slave-6kf3 krb5kdc[22177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:02 dist-test-slave-6kf3 krb5kdc[22177](info): set up 2 sockets
Sep 24 19:41:02 dist-test-slave-6kf3 krb5kdc[22177](info): commencing operation
krb5kdc: starting...
W20260924 19:41:03.231784 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:03 dist-test-slave-6kf3 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:03.342357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:03.421325 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:03.421532 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:03.421562 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:03.422696 22190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:03.422739 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:03.422757 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:03.422773 22190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:03.422788 22190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:03.424332 22190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:03.424572 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:03.424782 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:03.429276 22196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:03.429337 22195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:03.429454 22190 server_base.cc:1034] running on GCE node
W20260924 19:41:03.429353 22198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:03.429720 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:03.429975 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:03.431131 22190 hybrid_clock.cc:648] HybridClock initialized: now 1790278863431108 us; error 47 us; skew 500 ppm
Sep 24 19:41:03 dist-test-slave-6kf3 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:03.436898 22190 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:03.437366 22190 webserver.cc:460] Webserver started at http://127.0.98.62:45867/ using document root <none> and password file <none>
I20260924 19:41:03.437587 22190 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:03.437638 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:03.437763 22190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:03.438818 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "ebe8079794724a5ba54161951beebd9c"
format_stamp: "Formatted at 2026-09-24 19:41:03 on dist-test-slave-6kf3"
server_key: "da2ce2419b5c0b4d360ba6553a992da1"
server_key_iv: "23360f005bf4faa3244440c513048ebb"
server_key_version: "encryptionkey@0"
I20260924 19:41:03.439144 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebe8079794724a5ba54161951beebd9c"
format_stamp: "Formatted at 2026-09-24 19:41:03 on dist-test-slave-6kf3"
server_key: "da2ce2419b5c0b4d360ba6553a992da1"
server_key_iv: "23360f005bf4faa3244440c513048ebb"
server_key_version: "encryptionkey@0"
I20260924 19:41:03.440409 22190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:41:03.441087 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:03.441244 22190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:03.441324 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "ebe8079794724a5ba54161951beebd9c"
format_stamp: "Formatted at 2026-09-24 19:41:03 on dist-test-slave-6kf3"
server_key: "da2ce2419b5c0b4d360ba6553a992da1"
server_key_iv: "23360f005bf4faa3244440c513048ebb"
server_key_version: "encryptionkey@0"
I20260924 19:41:03.441381 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:03.444552 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:03.446559 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:03.446682 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:03.450248 22190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34157
I20260924 19:41:03.450274 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34157 every 8 connection(s)
I20260924 19:41:03.450565 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:03.451502 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:03.453727 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c: Bootstrap starting.
I20260924 19:41:03.454269 22258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:03.454512 22258 log.cc:824] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:03.455221 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c: No bootstrap required, opened a new log
I20260924 19:41:03.456799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22190
I20260924 19:41:03.456910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:03.457049 392 external_mini_cluster.cc:1428] Setting key f006c86bb17621671c218c7f10b3078b
I20260924 19:41:03.457079 22258 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 19:41:03.457245 22258 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:03.457294 22258 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe8079794724a5ba54161951beebd9c, State: Initialized, Role: FOLLOWER
I20260924 19:41:03.457648 22258 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 19:41:03.457731 22258 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:03.457765 22258 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:03.457818 22258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:03.458822 22258 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 19:41:03.458925 22258 leader_election.cc:302] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe8079794724a5ba54161951beebd9c; no voters:
I20260924 19:41:03.459108 22258 leader_election.cc:288] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:03.459127 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:03.457515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52711 (local address 127.0.98.62:34157)
0924 19:41:03.457704 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:03.457714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:03.457738 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:41:03.457850 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:03.457857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:03.457888 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:03.458014 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:03.458722 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:03.458725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:03.458726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:03.458729 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:03.458766 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:03.458776 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:03.458829 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:03.458894 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:41:03.458936 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 19:41:03.459151 22263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:03.459261 22263 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [term 1 LEADER]: Becoming Leader. State: Replica: ebe8079794724a5ba54161951beebd9c, State: Running, Role: LEADER
I20260924 19:41:03.459364 22263 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 19:41:03.459573 22258 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:03.459815 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe8079794724a5ba54161951beebd9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } } }
I20260924 19:41:03.460027 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:03.460063 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:03.460208 22269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:03.460637 22269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:03.459955 22265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe8079794724a5ba54161951beebd9c. Latest consensus state: current_term: 1 leader_uuid: "ebe8079794724a5ba54161951beebd9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe8079794724a5ba54161951beebd9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } } }
I20260924 19:41:03.461444 22265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:03.463099 22269 catalog_manager.cc:1347] Generated new cluster ID: 72b672884f7845fda966b671cad265cf
I20260924 19:41:03.463198 22269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:03.463263 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:03.461749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38909 (local address 127.0.98.62:34157)
0924 19:41:03.461780 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:03.461781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:03.462242 (+ 461us) server_negotiation.cc:407] Connection header received
0924 19:41:03.462306 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:03.462307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:03.462312 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:03.462366 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:03.462938 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:03.462939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:03.462940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:03.462944 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:03.462960 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:03.462965 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:03.463016 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:03.463154 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:41:03.463168 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:41:03.484390 22269 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:03.484584 22269 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:03.491214 22269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe8079794724a5ba54161951beebd9c: Generated new TSK 0
I20260924 19:41:03.491408 22269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:03.498999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22190
2026-09-24T19:41:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:03 dist-test-slave-6kf3 krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:03 dist-test-slave-6kf3 krb5kdc[22289](info): set up 2 sockets
Sep 24 19:41:03 dist-test-slave-6kf3 krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260924 19:41:04.443771 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:04.554864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:04.633003 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:04.633211 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:04.633241 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:04.634377 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:04.634423 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:04.634441 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:04.634457 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:04.634472 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:04.635916 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:04.636169 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:04.636368 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:04.640899 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:04.640950 22302 server_base.cc:1034] running on GCE node
W20260924 19:41:04.640878 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:04.640887 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:04.641373 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:04.641645 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:04.642781 22302 hybrid_clock.cc:648] HybridClock initialized: now 1790278864642770 us; error 30 us; skew 500 ppm
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:04.645146 22302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:04.645668 22302 webserver.cc:460] Webserver started at http://127.0.98.62:37291/ using document root <none> and password file <none>
I20260924 19:41:04.645910 22302 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:04.645970 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:04.646111 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:04.647539 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "45e173ca63f2448588602623ad15f6b7"
format_stamp: "Formatted at 2026-09-24 19:41:04 on dist-test-slave-6kf3"
server_key: "caced996b7c2bc99663da835ee64c37a"
server_key_iv: "8f5102cf22f4b08e765d4815642c1296"
server_key_version: "encryptionkey@0"
I20260924 19:41:04.647977 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "45e173ca63f2448588602623ad15f6b7"
format_stamp: "Formatted at 2026-09-24 19:41:04 on dist-test-slave-6kf3"
server_key: "caced996b7c2bc99663da835ee64c37a"
server_key_iv: "8f5102cf22f4b08e765d4815642c1296"
server_key_version: "encryptionkey@0"
I20260924 19:41:04.649591 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:41:04.650418 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:04.650629 22302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:41:04.650731 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "45e173ca63f2448588602623ad15f6b7"
format_stamp: "Formatted at 2026-09-24 19:41:04 on dist-test-slave-6kf3"
server_key: "caced996b7c2bc99663da835ee64c37a"
server_key_iv: "8f5102cf22f4b08e765d4815642c1296"
server_key_version: "encryptionkey@0"
I20260924 19:41:04.650804 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:04.659330 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:04.661409 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:04.661535 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:04.665156 22302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41085
I20260924 19:41:04.665186 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41085 every 8 connection(s)
I20260924 19:41:04.665485 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:04.666250 22370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:04.668433 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7: Bootstrap starting.
I20260924 19:41:04.668880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22302
I20260924 19:41:04.668990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:04.669018 22370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:04.669194 392 external_mini_cluster.cc:1428] Setting key e0e4f3bc9de896b34c17821fc44ee950
I20260924 19:41:04.669308 22370 log.cc:824] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:04.670010 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7: No bootstrap required, opened a new log
I20260924 19:41:04.672183 22370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } }
I20260924 19:41:04.672351 22370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:04.672397 22370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e173ca63f2448588602623ad15f6b7, State: Initialized, Role: FOLLOWER
I20260924 19:41:04.672516 22370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } }
I20260924 19:41:04.672595 22370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:04.672628 22370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:04.672670 22370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:04.673277 22370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } }
I20260924 19:41:04.673362 22370 leader_election.cc:302] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e173ca63f2448588602623ad15f6b7; no voters:
I20260924 19:41:04.673511 22370 leader_election.cc:288] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:04.673558 22375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:04.673734 22370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:04.673744 22375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 45e173ca63f2448588602623ad15f6b7, State: Running, Role: LEADER
I20260924 19:41:04.673853 22375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } }
I20260924 19:41:04.674186 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e173ca63f2448588602623ad15f6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } } }
I20260924 19:41:04.674255 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:04.674410 22377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e173ca63f2448588602623ad15f6b7. Latest consensus state: current_term: 1 leader_uuid: "45e173ca63f2448588602623ad15f6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e173ca63f2448588602623ad15f6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41085 } } }
I20260924 19:41:04.674469 22377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:04.674577 22384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:04.675050 22384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:04.676818 22384 catalog_manager.cc:1347] Generated new cluster ID: 7caa1a8c2cb54e7884274e6087eb889c
I20260924 19:41:04.676864 22384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:04.679863 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:04.669676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57549 (local address 127.0.98.62:41085)
0924 19:41:04.669807 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:04.669815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:04.669846 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:41:04.670289 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:04.670295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:04.670341 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:04.670465 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:04.671238 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:04.672139 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:04.673171 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:04.673712 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:04.674136 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:04.676428 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:04.676431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:04.676431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:04.676441 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:04.678323 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:04.678730 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:04.678733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:04.678733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:04.678827 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:04.679006 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:04.679007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:04.679007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:04.679278 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:04.679417 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:04.679529 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:41:04.679659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260924 19:41:04.681063 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:04.683774 22384 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:04.684428 22384 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:04.689262 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:04.682443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58811 (local address 127.0.98.62:41085)
0924 19:41:04.682481 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:04.682482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:04.682552 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:41:04.682637 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:04.682638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:04.682647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:04.682673 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:04.683319 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:04.683976 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:04.685055 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:04.685354 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:04.685386 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:04.686811 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:04.686813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:04.686813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:04.686819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:04.688251 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:04.688601 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:04.688602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:04.688602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:04.688650 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:04.688812 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:04.688813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:04.688813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:04.688918 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:04.688997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:04.689097 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:41:04.689174 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:41:04.693398 22384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e173ca63f2448588602623ad15f6b7: Generated new TSK 0
I20260924 19:41:04.693588 22384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:04.697834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22302
2026-09-24T19:41:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22401](info): set up 2 sockets
Sep 24 19:41:04 dist-test-slave-6kf3 krb5kdc[22401](info): commencing operation
krb5kdc: starting...
W20260924 19:41:05.647749 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:05 dist-test-slave-6kf3 krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:05.771148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:05.848733 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:05.848937 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:05.848959 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:05.850028 22414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:05.850059 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:05.850071 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:05.850090 22414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:05.850100 22414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:05.851495 22414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:05.851686 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:05.851861 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:05.856448 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:05.856539 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:05.856596 22414 server_base.cc:1034] running on GCE node
W20260924 19:41:05.856607 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:05.856935 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:05.857177 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:05.858307 22414 hybrid_clock.cc:648] HybridClock initialized: now 1790278865858290 us; error 34 us; skew 500 ppm
Sep 24 19:41:05 dist-test-slave-6kf3 krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:05.860724 22414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:05.861156 22414 webserver.cc:460] Webserver started at http://127.0.98.62:39421/ using document root <none> and password file <none>
I20260924 19:41:05.861366 22414 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:05.861414 22414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:05.861534 22414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:05.862532 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "e0bbad1929f14187ab311fd13c5452b7"
format_stamp: "Formatted at 2026-09-24 19:41:05 on dist-test-slave-6kf3"
server_key: "744ef4324150e95f6cd59ef110598eeb"
server_key_iv: "6dde6b357560cf8bd8ef953af4c848b7"
server_key_version: "encryptionkey@0"
I20260924 19:41:05.862855 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0bbad1929f14187ab311fd13c5452b7"
format_stamp: "Formatted at 2026-09-24 19:41:05 on dist-test-slave-6kf3"
server_key: "744ef4324150e95f6cd59ef110598eeb"
server_key_iv: "6dde6b357560cf8bd8ef953af4c848b7"
server_key_version: "encryptionkey@0"
I20260924 19:41:05.864069 22414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:41:05.864738 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:05.864892 22414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:05.864979 22414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "e0bbad1929f14187ab311fd13c5452b7"
format_stamp: "Formatted at 2026-09-24 19:41:05 on dist-test-slave-6kf3"
server_key: "744ef4324150e95f6cd59ef110598eeb"
server_key_iv: "6dde6b357560cf8bd8ef953af4c848b7"
server_key_version: "encryptionkey@0"
I20260924 19:41:05.865046 22414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:05.872027 22414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:05.873994 22414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:05.874127 22414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:05.877763 22414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44897
I20260924 19:41:05.877792 22481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44897 every 8 connection(s)
I20260924 19:41:05.878103 22414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:05.878909 22482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:05.881057 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7: Bootstrap starting.
I20260924 19:41:05.881551 22482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:05.881785 22482 log.cc:824] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:05.882359 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7: No bootstrap required, opened a new log
I20260924 19:41:05.883847 22482 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } }
I20260924 19:41:05.884022 22482 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:05.884053 22482 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bbad1929f14187ab311fd13c5452b7, State: Initialized, Role: FOLLOWER
I20260924 19:41:05.884157 22482 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } }
I20260924 19:41:05.884207 22482 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:05.884234 22482 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:05.884271 22482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:05.884716 22482 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } }
I20260924 19:41:05.884792 22482 leader_election.cc:302] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0bbad1929f14187ab311fd13c5452b7; no voters:
I20260924 19:41:05.884903 22482 leader_election.cc:288] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:05.884938 22485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:05.885113 22482 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:05.885125 22485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [term 1 LEADER]: Becoming Leader. State: Replica: e0bbad1929f14187ab311fd13c5452b7, State: Running, Role: LEADER
I20260924 19:41:05.885259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22414
I20260924 19:41:05.885293 22485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } }
I20260924 19:41:05.885365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:05.885546 392 external_mini_cluster.cc:1428] Setting key 5e64de186b7ac37546ffb4db3a73a4c1
I20260924 19:41:05.885746 22487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0bbad1929f14187ab311fd13c5452b7. Latest consensus state: current_term: 1 leader_uuid: "e0bbad1929f14187ab311fd13c5452b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } } }
I20260924 19:41:05.885823 22487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:05.885957 22486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0bbad1929f14187ab311fd13c5452b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bbad1929f14187ab311fd13c5452b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44897 } } }
I20260924 19:41:05.886014 22486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:05.886394 22490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:05.886912 22490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:05.889339 22490 catalog_manager.cc:1347] Generated new cluster ID: ef099b54c59c4a78b9f022c6038ffc4e
I20260924 19:41:05.889405 22490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:05.891294 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:05.886170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53479 (local address 127.0.98.62:44897)
0924 19:41:05.886487 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:05.886498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:05.886534 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:41:05.886656 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:05.886663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:05.886721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:05.886870 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:05.887102 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:05.888154 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:05.889290 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:05.889887 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:05.889987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:05.890671 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:05.890675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:05.890676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:05.890864 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:05.890899 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:05.890913 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:05.890971 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:05.891042 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 19:41:05.891099 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":78,"threads_started":1}
I20260924 19:41:05.892139 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:05.896723 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:05.893530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41359 (local address 127.0.98.62:44897)
0924 19:41:05.893570 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:05.893572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:05.893710 (+ 138us) server_negotiation.cc:407] Connection header received
0924 19:41:05.893755 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:05.893755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:05.893765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:05.893807 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:05.893922 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:05.894593 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:05.895552 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:05.895832 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:05.895884 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:05.896506 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:05.896508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:05.896508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:05.896512 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:05.896524 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:05.896528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:05.896568 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:05.896633 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:41:05.896655 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:41:05.918476 22490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:05.919257 22490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:05.925535 22490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0bbad1929f14187ab311fd13c5452b7: Generated new TSK 0
I20260924 19:41:05.925722 22490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:05.931480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22414
2026-09-24T19:41:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:05 dist-test-slave-6kf3 krb5kdc[22513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:05 dist-test-slave-6kf3 krb5kdc[22513](info): set up 2 sockets
Sep 24 19:41:05 dist-test-slave-6kf3 krb5kdc[22513](info): commencing operation
krb5kdc: starting...
W20260924 19:41:06.879794 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:06 dist-test-slave-6kf3 krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:06.990799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:07.070307 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:07.070518 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:07.070549 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:07.071668 22526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:07.071710 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:07.071728 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:07.071745 22526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:07.071760 22526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:07.073217 22526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:07.073449 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:07.073660 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:07.078198 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:07.078245 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:07.078222 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:07.078293 22526 server_base.cc:1034] running on GCE node
I20260924 19:41:07.078637 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:07.078913 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:07.080049 22526 hybrid_clock.cc:648] HybridClock initialized: now 1790278867080031 us; error 26 us; skew 500 ppm
Sep 24 19:41:07 dist-test-slave-6kf3 krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:07.082510 22526 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:07.082930 22526 webserver.cc:460] Webserver started at http://127.0.98.62:35911/ using document root <none> and password file <none>
I20260924 19:41:07.083145 22526 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:07.083192 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:07.083310 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:07.084388 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "561c8eac794443b1a9741275ee338ea4"
format_stamp: "Formatted at 2026-09-24 19:41:07 on dist-test-slave-6kf3"
server_key: "9ebfa22b13deab4fecb75ed1b685abe3"
server_key_iv: "a8c1e565471c1b7a058eb67af98e0403"
server_key_version: "encryptionkey@0"
I20260924 19:41:07.084712 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "561c8eac794443b1a9741275ee338ea4"
format_stamp: "Formatted at 2026-09-24 19:41:07 on dist-test-slave-6kf3"
server_key: "9ebfa22b13deab4fecb75ed1b685abe3"
server_key_iv: "a8c1e565471c1b7a058eb67af98e0403"
server_key_version: "encryptionkey@0"
I20260924 19:41:07.085951 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:41:07.086647 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:07.086794 22526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:07.086874 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "561c8eac794443b1a9741275ee338ea4"
format_stamp: "Formatted at 2026-09-24 19:41:07 on dist-test-slave-6kf3"
server_key: "9ebfa22b13deab4fecb75ed1b685abe3"
server_key_iv: "a8c1e565471c1b7a058eb67af98e0403"
server_key_version: "encryptionkey@0"
I20260924 19:41:07.086931 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:07.090154 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:07.091959 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:07.092098 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:07.095659 22526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44281
I20260924 19:41:07.095691 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44281 every 8 connection(s)
I20260924 19:41:07.096007 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:07.096874 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:07.099048 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4: Bootstrap starting.
I20260924 19:41:07.099581 22594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:07.099849 22594 log.cc:824] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:07.100539 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4: No bootstrap required, opened a new log
I20260924 19:41:07.102073 22594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } }
I20260924 19:41:07.102211 22594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:41:07.102232 22594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561c8eac794443b1a9741275ee338ea4, State: Initialized, Role: FOLLOWER
I20260924 19:41:07.102352 22594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } }
I20260924 19:41:07.102416 22594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:41:07.102478 22594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:41:07.102525 22594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:07.103021 22594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } }
I20260924 19:41:07.103104 22594 leader_election.cc:302] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 561c8eac794443b1a9741275ee338ea4; no voters:
I20260924 19:41:07.103248 22594 leader_election.cc:288] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:41:07.103319 22597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:07.103482 22594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:07.103495 22597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [term 1 LEADER]: Becoming Leader. State: Replica: 561c8eac794443b1a9741275ee338ea4, State: Running, Role: LEADER
I20260924 19:41:07.103667 22597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } }
I20260924 19:41:07.104303 22599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561c8eac794443b1a9741275ee338ea4. Latest consensus state: current_term: 1 leader_uuid: "561c8eac794443b1a9741275ee338ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } } }
I20260924 19:41:07.104377 22599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:07.104467 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "561c8eac794443b1a9741275ee338ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561c8eac794443b1a9741275ee338ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44281 } } }
I20260924 19:41:07.104571 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:07.104635 22607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:07.105253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22526
I20260924 19:41:07.105304 22607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:07.105362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:07.105520 392 external_mini_cluster.cc:1428] Setting key b495880139f48165c69d74fb9caf81c9
I20260924 19:41:07.106763 22607 catalog_manager.cc:1347] Generated new cluster ID: de6a91f0c8e94983b39500c9b5ad86be
I20260924 19:41:07.106812 22607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:07.117601 22607 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:07.117769 22607 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:07.117784 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:07.106174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49943 (local address 127.0.98.62:44281)
0924 19:41:07.116070 (+ 9896us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:07.116080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:07.116121 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:41:07.116241 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:07.116247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:07.116274 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:07.116446 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:07.117257 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:07.117262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:07.117262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:07.117267 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:07.117325 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:07.117342 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:07.117415 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:07.117494 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:41:07.117543 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9834,"thread_start_us":82,"threads_started":1}
I20260924 19:41:07.118813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:07.121580 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:07.120501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57271 (local address 127.0.98.62:44281)
0924 19:41:07.120550 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:07.120552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:07.120641 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:41:07.120721 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:07.120722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:07.120726 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:07.120780 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:07.121368 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:07.121369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:07.121370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:41:07.121373 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:07.121388 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:41:07.121392 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:07.121439 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:07.121505 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:41:07.121516 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:41:07.124485 22607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 561c8eac794443b1a9741275ee338ea4: Generated new TSK 0
I20260924 19:41:07.124653 22607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:07.129204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22526
2026-09-24T19:41:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1197 ms)
[----------] 18 tests from AuthTokenIssuingTest (21842 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:07 dist-test-slave-6kf3 krb5kdc[22625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:07 dist-test-slave-6kf3 krb5kdc[22625](info): set up 2 sockets
Sep 24 19:41:07 dist-test-slave-6kf3 krb5kdc[22625](info): commencing operation
krb5kdc: starting...
W20260924 19:41:08.063982 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:08.177029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:08.255167 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:08.255395 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:08.255412 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:08.255432 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:08.256628 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:08.256669 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:08.256683 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:08.256695 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:08.256705 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:08.258178 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:08.258411 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:08.258622 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:08.263336 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:08.263336 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:08.263347 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:08.263543 22638 server_base.cc:1034] running on GCE node
I20260924 19:41:08.263765 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:08.264061 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:08.265205 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790278868265190 us; error 28 us; skew 500 ppm
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:08.267607 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:08.268038 22638 webserver.cc:460] Webserver started at http://127.0.98.62:37313/ using document root <none> and password file <none>
I20260924 19:41:08.268213 22638 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:08.268252 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:08.268339 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:08.269341 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "a2ef5b5a2b314fd796f93b290fa2a125"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "e6ccd8828cf1050a2bc84261e83b4365"
server_key_iv: "021fa2e2840e5314a243f9e14e0a956e"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.269634 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2ef5b5a2b314fd796f93b290fa2a125"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "e6ccd8828cf1050a2bc84261e83b4365"
server_key_iv: "021fa2e2840e5314a243f9e14e0a956e"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.270876 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:41:08.271481 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:08.271637 22638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:08.271721 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "a2ef5b5a2b314fd796f93b290fa2a125"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "e6ccd8828cf1050a2bc84261e83b4365"
server_key_iv: "021fa2e2840e5314a243f9e14e0a956e"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.271793 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:08.286630 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:08.288652 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:08.288771 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:08.292800 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40955
I20260924 19:41:08.292843 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40955 every 8 connection(s)
I20260924 19:41:08.293177 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:08.294114 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:08.295768 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:08.301378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22638
I20260924 19:41:08.301496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-0/wal/instance
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:08.301671 392 external_mini_cluster.cc:1428] Setting key cce6f2a8a6db2f2001e2684bc211694f
I20260924 19:41:08.305109 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.296599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.62:40955)
0924 19:41:08.296707 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.296713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.296745 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:41:08.297287 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.297290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.297315 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.297390 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.297914 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.298734 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.299563 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.299873 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.299987 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.302119 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.302129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.302129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.302137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.303588 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.304009 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.304011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.304012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.304092 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.304306 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.304306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.304306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.304513 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.304579 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.304792 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.304909 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":41,"threads_started":1}
I20260924 19:41:08.305131 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.296438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.0.1:53440)
0924 19:41:08.296597 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.296632 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.297111 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.297400 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.297402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.297677 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.297827 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.297832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.298818 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.298819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.299490 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.299492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.299979 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.299981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.300043 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.300548 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.300557 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.302029 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.303668 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.303675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.303678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.303914 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.304150 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.304151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.304151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.304253 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.304573 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.304578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.304657 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.304753 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.304996 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":89,"thread_start_us":74,"threads_started":1}
I20260924 19:41:08.305800 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:08.306144 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.306011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34488)
0924 19:41:08.306059 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.306112 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:41:08.306285 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
I20260924 19:41:08.306689 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.306599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34496)
0924 19:41:08.306641 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.306668 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:41:08.306800 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:41:08.325153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32853
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:41:08.356025 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } attempt: 1
I20260924 19:41:08.356480 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.356367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34498)
0924 19:41:08.356417 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.356451 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:41:08.356850 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.356774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34506)
0924 19:41:08.356811 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.356834 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:41:08.356953 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
W20260924 19:41:08.404624 22713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:08.404832 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:08.404848 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:08.404867 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:08.405936 22713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:08.405970 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:08.405982 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:08.405993 22713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:08.406008 22713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:08.407454 22713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:08.407648 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:08.407826 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:08.412228 22718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:08.412431 22713 server_base.cc:1034] running on GCE node
W20260924 19:41:08.412281 22721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:08.412240 22719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:08.412767 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:08.413017 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:08.414163 22713 hybrid_clock.cc:648] HybridClock initialized: now 1790278868414149 us; error 27 us; skew 500 ppm
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:08.416539 22713 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:41:08.416931 22713 webserver.cc:460] Webserver started at http://127.0.98.61:46189/ using document root <none> and password file <none>
I20260924 19:41:08.417146 22713 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:08.417193 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:08.417315 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:08.418324 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data/instance:
uuid: "561bd622bb6b46df98331bc50b73318e"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "586648138c037ee5d010f6cfdae5b94e"
server_key_iv: "bba2cec838dc5cdcc364e1ebd2d338c3"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.418642 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal/instance:
uuid: "561bd622bb6b46df98331bc50b73318e"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "586648138c037ee5d010f6cfdae5b94e"
server_key_iv: "bba2cec838dc5cdcc364e1ebd2d338c3"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.419978 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:41:08.420673 22728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:08.420835 22713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:08.420917 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "561bd622bb6b46df98331bc50b73318e"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "586648138c037ee5d010f6cfdae5b94e"
server_key_iv: "bba2cec838dc5cdcc364e1ebd2d338c3"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.420977 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:08.425128 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } attempt: 2
I20260924 19:41:08.425504 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.425378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34512)
0924 19:41:08.425430 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.425470 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:41:08.425834 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.425748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34514)
0924 19:41:08.425786 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.425813 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:41:08.425915 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32853: connect: Connection refused (error 111)
I20260924 19:41:08.442173 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:08.444352 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:08.444463 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:08.448180 22713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32853
I20260924 19:41:08.448248 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32853 every 8 connection(s)
I20260924 19:41:08.448560 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:41:08.449527 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:08.449674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22713
I20260924 19:41:08.449764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-1/wal/instance
I20260924 19:41:08.449919 392 external_mini_cluster.cc:1428] Setting key 724c6239a62954cffa3adce5f0cf9364
I20260924 19:41:08.451110 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:08.461566 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.0.98.62:40955)
0924 19:41:08.451845 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.451847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.452073 (+ 226us) server_negotiation.cc:407] Connection header received
0924 19:41:08.452552 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.452553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.452568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.452619 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.453273 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.453745 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.454674 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.454869 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.454923 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.458758 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.458760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.458760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.458769 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.460248 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.460708 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.460708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.460709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.460754 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.460977 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.460977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.460978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.461075 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.461130 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.461433 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.461498 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:41:08.461824 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.451763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.0.1:53452)
0924 19:41:08.451924 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.451989 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.452462 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.452696 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.452701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.452989 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.453189 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.453193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.453823 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.453824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.454589 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.454591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.454953 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.454954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.455075 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.455531 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.455539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.458664 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.460321 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.460328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.460330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.460635 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.460803 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.460803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.460804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.460926 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.461124 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.461129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.461295 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.461414 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.461591 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260924 19:41:08.462361 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:41:08.470459 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.462587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34520)
0924 19:41:08.462641 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.462650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.462687 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.462981 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.462982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.463084 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.463134 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.463135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.464358 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.464359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.464848 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.464849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.465446 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.465446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.465494 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.465842 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.465845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.468190 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.469575 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.469576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.469577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.469770 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.469933 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.469933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.469933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.469978 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.470206 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.470206 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.470244 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.470295 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.470372 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:41:08.470530 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.462650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.0.98.61:32853)
0924 19:41:08.462825 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.462831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.462852 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:41:08.462875 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.462878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.462892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.462966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.463195 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.464294 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.464922 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.465378 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.465453 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.468264 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.468271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.468272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.468278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.469506 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.469828 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.469831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.469832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.469888 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.470018 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.470018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.470018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.470154 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.470214 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.470329 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.470418 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260924 19:41:08.471099 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:08.471400 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.471303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40434)
0924 19:41:08.471341 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.471382 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:41:08.471508 22730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260924 19:41:08.471899 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.471820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40448)
0924 19:41:08.471860 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.471884 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:41:08.471997 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:41:08.474447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42257
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:41:08.517050 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } attempt: 3
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:41:08.521191 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 1
I20260924 19:41:08.521538 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.521432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40450)
0924 19:41:08.521484 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.521509 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:41:08.521943 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.521754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40454)
0924 19:41:08.521887 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.521912 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260924 19:41:08.522023 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260924 19:41:08.523919 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.517437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34530)
0924 19:41:08.517495 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.517508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.517563 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.517695 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.517696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.517826 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.517888 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.517890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.518791 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.518791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.519305 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.519306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.519758 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.519758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.519808 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.520233 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.520238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.521392 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.523044 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.523045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.523046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.523223 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.523414 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.523414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.523415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.523464 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.523696 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.523697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.523736 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.523784 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.523857 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:41:08.523993 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.517489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.0.98.61:32853)
0924 19:41:08.517567 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.517570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.517593 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:41:08.517625 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.517626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.517639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.517689 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.517952 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.518719 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.519371 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.519698 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.519763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.521466 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.521467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.521467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.521475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.522964 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.523279 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.523280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.523280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.523339 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.523512 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.523513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.523513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.523647 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.523706 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.523810 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.523890 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:41:08.524403 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:08.524716 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.524610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40460)
0924 19:41:08.524654 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.524687 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:41:08.525007 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.524928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40474)
0924 19:41:08.524963 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.524988 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 19:41:08.525094 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
W20260924 19:41:08.554311 22788 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:08.554522 22788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:08.554548 22788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:08.554574 22788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:08.555698 22788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:08.555737 22788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:08.555755 22788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:08.555771 22788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:08.555786 22788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:08.556243 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 1
I20260924 19:41:08.556661 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.556513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40478)
0924 19:41:08.556577 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.556626 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:41:08.557026 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.556941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40488)
0924 19:41:08.556980 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.557004 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:41:08.557116 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260924 19:41:08.557510 22788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40955,127.0.98.61:32853,127.0.98.60:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:08.557703 22788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:08.557874 22788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:08.562404 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:08.562477 22788 server_base.cc:1034] running on GCE node
W20260924 19:41:08.562618 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:08.562388 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:08.562871 22788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:08.563138 22788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:08.564273 22788 hybrid_clock.cc:648] HybridClock initialized: now 1790278868564259 us; error 24 us; skew 500 ppm
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:08.566540 22788 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:41:08.566901 22788 webserver.cc:460] Webserver started at http://127.0.98.60:45939/ using document root <none> and password file <none>
I20260924 19:41:08.567086 22788 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:08.567127 22788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:08.567214 22788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:08.568251 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data/instance:
uuid: "170101d7e7d64d6cb87014c7bf965b79"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "5cf5240f8c44e448dd13298d80ddd233"
server_key_iv: "5d4ab22141f46620bda26f9f68c87a21"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.568534 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal/instance:
uuid: "170101d7e7d64d6cb87014c7bf965b79"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "5cf5240f8c44e448dd13298d80ddd233"
server_key_iv: "5d4ab22141f46620bda26f9f68c87a21"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.569662 22788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:41:08.570264 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:08.570421 22788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:08.570497 22788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "170101d7e7d64d6cb87014c7bf965b79"
format_stamp: "Formatted at 2026-09-24 19:41:08 on dist-test-slave-6kf3"
server_key: "5cf5240f8c44e448dd13298d80ddd233"
server_key_iv: "5d4ab22141f46620bda26f9f68c87a21"
server_key_version: "encryptionkey@0"
I20260924 19:41:08.570547 22788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:08.590160 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 2
I20260924 19:41:08.590581 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.590470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40500)
0924 19:41:08.590524 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.590552 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:41:08.590934 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.590859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40510)
0924 19:41:08.590896 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.590918 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:41:08.591020 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42257: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42257: connect: Connection refused (error 111)
I20260924 19:41:08.592183 22788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:08.594326 22788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:08.594461 22788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:08.598213 22788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42257
I20260924 19:41:08.598244 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42257 every 8 connection(s)
I20260924 19:41:08.598578 22788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:41:08.599519 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:08.601079 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:08.608778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 22788
I20260924 19:41:08.608879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278584646480-392-0/minicluster-data/master-2/wal/instance
I20260924 19:41:08.609074 392 external_mini_cluster.cc:1428] Setting key 76df0e25a66ece62f73903a7aaf7f819
I20260924 19:41:08.609268 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.601729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.62:40955)
0924 19:41:08.601804 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.601806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.601966 (+ 160us) server_negotiation.cc:407] Connection header received
0924 19:41:08.602403 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.602404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.602413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.602460 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.603034 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.603486 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.604445 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.604636 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.604727 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.606778 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.606780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.606780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.606787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.608030 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.608384 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.608385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.608386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.608439 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.608616 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.608616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.608617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.608715 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.608757 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.609108 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.609186 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:41:08.609442 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.601695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.0.1:53464)
0924 19:41:08.601845 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.601883 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.602329 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.602513 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.602516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.602785 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.602974 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.602978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.603562 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.603563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.604378 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.604380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.604776 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.604777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.604888 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.605374 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.605382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.606688 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.608096 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.608101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.608104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.608308 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.608490 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.608490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.608490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.608574 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.608764 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.608770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.608965 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.609076 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.609237 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 19:41:08.610030 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:08.616196 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.610320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.98.61:32853)
0924 19:41:08.610366 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.610368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.610375 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:41:08.610401 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.610402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.610413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.610469 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.610711 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.611168 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.611811 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.612121 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.612167 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.613880 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.613882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.613882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.613889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.615160 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.615510 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.615511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.615512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.615567 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.615742 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.615743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.615743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.615838 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.615886 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.616078 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.616135 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:41:08.616217 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.610244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32853 (local address 127.0.0.1:34546)
0924 19:41:08.610279 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.610289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.610323 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.610472 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.610473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.610604 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.610646 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.610647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.611227 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.611228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.611744 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.611746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.612173 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.612174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.612226 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.612601 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.612605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.613819 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.615229 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.615230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.615230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.615448 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.615631 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.615632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.615632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.615689 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.615895 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.615896 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.615959 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.616034 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.616121 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 19:41:08.616520 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:08.617697 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.609447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.0.98.60:42257)
0924 19:41:08.609742 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.609748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.609772 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:41:08.609793 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.609795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.609807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.609874 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.610498 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.611274 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.612008 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.612379 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.612434 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.614779 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.614787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.614787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.614793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.616469 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.616765 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.616768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.616769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.616823 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.617202 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.617202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.617202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.617318 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.617466 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.617533 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.617600 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:08.623294 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.616755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40524)
0924 19:41:08.616792 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.616800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.616830 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.617029 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.617029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.617121 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.617149 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.617150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.617937 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.617938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.618524 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.618526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.618859 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.618860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.618914 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.619223 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.619228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.620299 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.622343 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.622344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.622344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.622532 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.622704 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.622705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.622706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.622765 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.622994 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.622995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.623036 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.623138 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.623231 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:41:08.623466 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.616788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.98.60:42257)
0924 19:41:08.616923 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.616925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.616930 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:41:08.616945 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.616945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.616953 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.617014 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.617203 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.617658 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.618585 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.618795 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.618841 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.620367 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.620368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.620368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.620375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.622268 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.622600 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.622600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.622601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.622648 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.622816 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.622816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.622817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.622929 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.622991 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.623169 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.623288 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260924 19:41:08.625191 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: Bootstrap starting.
I20260924 19:41:08.625272 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.618431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.0.98.62:40955)
0924 19:41:08.618467 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.618469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.618520 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:41:08.618598 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.618599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.618609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.618642 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.619181 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.619662 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.620401 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.620587 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.620644 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.622788 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.622789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.622790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.622796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.624288 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.624653 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.624654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.624655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.624707 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.624876 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.624876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.624876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.624966 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.625075 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.625151 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.625203 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:41:08.625886 22857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:08.626225 22857 log.cc:824] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:08.626942 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: No bootstrap required, opened a new log
I20260924 19:41:08.628834 22857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.629009 22857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170101d7e7d64d6cb87014c7bf965b79, State: Initialized, Role: FOLLOWER
I20260924 19:41:08.629110 22857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.629376 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.629446 22857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:08.629458 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260924 19:41:08.631765 22875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:41:08.631866 22875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:41:08.632243 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 2
I20260924 19:41:08.633332 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.625917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.98.61:32853)
0924 19:41:08.625955 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.625957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.625965 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:41:08.626007 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.626008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.626017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.626062 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.626567 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.627085 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.627907 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.628125 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.628170 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.630537 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.630539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.630539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.630546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.632129 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.632495 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.632496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.632497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.632550 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.632732 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.632733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.632733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.632849 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.632966 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.633175 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.633248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:41:08.631577 22788 master_runner.cc:186] Error getting master registration for 127.0.98.62:40955: OK, instance_id { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" instance_seqno: 1790278868291492 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:41:08.633749 22788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:41:08.634924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:08.635898 22680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a2ef5b5a2b314fd796f93b290fa2a125"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53478
I20260924 19:41:08.639495 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.632704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.0.98.60:42257)
0924 19:41:08.632737 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.632739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.632744 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:41:08.632761 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.632762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.632770 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.632810 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.633125 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.633783 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.635008 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.635293 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.635346 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.637043 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.637052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.637053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.637059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.638496 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.638795 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.638796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.638796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.638847 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.639000 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.639001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.639001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.639158 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.639216 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.639348 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.639413 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:41:08.639500 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.632480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40538)
0924 19:41:08.632587 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.632598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.632633 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.632844 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.632845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.632991 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.633034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.633035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.633848 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.633849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.634922 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.634924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.635395 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.635396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.635452 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.635857 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.635862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.636971 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.638574 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.638575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.638576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.638737 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.638897 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.638897 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.638898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.638952 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.639219 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.639220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.639262 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.639320 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.639408 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 19:41:08.640672 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: Bootstrap starting.
I20260924 19:41:08.641319 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:08.641601 22706 log.cc:824] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:08.642364 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: No bootstrap required, opened a new log
I20260924 19:41:08.643298 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.643400 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ef5b5a2b314fd796f93b290fa2a125, State: Initialized, Role: FOLLOWER
I20260924 19:41:08.643468 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.643697 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.643754 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:08.643772 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.643882 22680 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:41:08.643972 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:08.644690 22680 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.644918 22680 leader_election.cc:288] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [CANDIDATE]: Term 1 election: Requested vote from peers 561bd622bb6b46df98331bc50b73318e (127.0.98.61:32853), 170101d7e7d64d6cb87014c7bf965b79 (127.0.98.60:42257)
I20260924 19:41:08.652354 22831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "170101d7e7d64d6cb87014c7bf965b79"
I20260924 19:41:08.652494 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:08.650198 22638 master_runner.cc:186] Error getting master registration for 127.0.98.61:32853: OK, instance_id { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" instance_seqno: 1790278868447048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:41:08.652897 22638 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:41:08.645462 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:41:08.653164 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:41:08.653434 22831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ef5b5a2b314fd796f93b290fa2a125 in term 1.
I20260924 19:41:08.653677 22655 leader_election.cc:302] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170101d7e7d64d6cb87014c7bf965b79, a2ef5b5a2b314fd796f93b290fa2a125; no voters:
I20260924 19:41:08.653786 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:08.653932 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [term 1 LEADER]: Becoming Leader. State: Replica: a2ef5b5a2b314fd796f93b290fa2a125, State: Running, Role: LEADER
I20260924 19:41:08.654027 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.654511 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ef5b5a2b314fd796f93b290fa2a125. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.654611 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:08.655572 22897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:08.655759 22897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:08.664050 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.655072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.61:32853)
0924 19:41:08.655111 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.655113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.655439 (+ 326us) server_negotiation.cc:407] Connection header received
0924 19:41:08.655580 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.655581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.655604 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.655634 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.656829 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.657521 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.658512 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.659249 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.659307 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.661093 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.661094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.661094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.661101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.662985 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.663322 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.663323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.663323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.663377 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.663533 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.663533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.663534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.663674 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.663779 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.663869 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.663958 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:41:08.670966 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.653752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.0.98.62:40955)
0924 19:41:08.654459 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.654461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.661824 (+ 7363us) server_negotiation.cc:407] Connection header received
0924 19:41:08.662558 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.662560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.662571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.662599 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.664444 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.665095 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.666007 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.666294 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.666349 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.668076 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.668077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.668077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.668084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.669643 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.670187 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.670189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.670189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.670245 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.670430 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.670431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.670431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.670548 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.670635 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.670806 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.670877 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684}
I20260924 19:41:08.671041 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.654901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.0.98.60:42257)
0924 19:41:08.654937 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.654939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.656893 (+ 1954us) server_negotiation.cc:407] Connection header received
0924 19:41:08.656974 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.656975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.656985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.657013 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.658958 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.659659 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.661874 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.662165 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.662206 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.666475 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.666476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.666477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.666483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.668013 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.668931 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.668933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.668933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.669001 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.669890 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.669891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.669891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.670137 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.670378 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.670723 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.670889 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:41:08.674796 22831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [term 1 FOLLOWER]: Refusing update from remote peer a2ef5b5a2b314fd796f93b290fa2a125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:41:08.676646 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:41:08.678221 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ef5b5a2b314fd796f93b290fa2a125. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.678313 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.680063 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.680156 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:08.680338 22901 mvcc.cc:205] Tried to move back new op lower bound from 7332982246061293568 to 7332982246007586816. Current Snapshot: MvccSnapshot[applied={T|T < 7332982246061293568}]
I20260924 19:41:08.680356 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.680420 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.680637 22902 mvcc.cc:205] Tried to move back new op lower bound from 7332982246061293568 to 7332982246007586816. Current Snapshot: MvccSnapshot[applied={T|T < 7332982246061293568}]
I20260924 19:41:08.681167 22897 catalog_manager.cc:1347] Generated new cluster ID: 98f2ba09c2dc459cbe361ae3f230d3bf
I20260924 19:41:08.681213 22897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:08.682153 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } attempt: 3
Sep 24 19:41:08 dist-test-slave-6kf3 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:08.691475 22897 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:08.692142 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.682384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42257 (local address 127.0.0.1:40556)
0924 19:41:08.682458 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:08.682469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:08.682502 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:08.682824 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:08.682824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:08.682991 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:08.683034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.683035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.683985 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.683986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.684761 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:08.684762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.685252 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.685253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:08.685317 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.685888 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:08.685894 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:08.687392 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:08.690432 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.690433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.690434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.690660 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.691330 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:08.691331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:08.691331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:08.691400 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:08.691800 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:08.691801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:08.691858 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 19:41:08.691911 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:41:08.692034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:41:08.692557 22897 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:41:08.692700 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:08.682588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.98.60:42257)
0924 19:41:08.682692 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.682694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.682701 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:41:08.682734 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.682735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.682743 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.682783 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.683114 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.683863 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.684850 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:08.685142 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:08.685190 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:08.687491 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:08.687492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:08.687493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:08.687500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:08.689953 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.690739 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.690740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.690741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.691264 (+ 523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:08.691465 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:08.691466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:08.691467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:08.691688 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:08.692319 (+ 631us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:08.692388 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:41:08.692520 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 19:41:08.694219 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: Bootstrap starting.
I20260924 19:41:08.695009 22781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:08.695297 22781 log.cc:824] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:08.706473 22897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125: Generated new TSK 0
I20260924 19:41:08.706862 22897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:08.708719 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: No bootstrap required, opened a new log
I20260924 19:41:08.710731 22781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.710880 22781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561bd622bb6b46df98331bc50b73318e, State: Initialized, Role: FOLLOWER
I20260924 19:41:08.710966 22781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } }
I20260924 19:41:08.711320 22781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:08.711418 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "561bd622bb6b46df98331bc50b73318e"
I20260924 19:41:08.711501 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:08.712531 22755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ef5b5a2b314fd796f93b290fa2a125 in term 1.
I20260924 19:41:08.712821 22754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [term 1 FOLLOWER]: Refusing update from remote peer a2ef5b5a2b314fd796f93b290fa2a125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:41:08.711262 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.713146 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.713223 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:41:08.713938 22919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:41:08.713991 22919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:41:08.716176 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ef5b5a2b314fd796f93b290fa2a125. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.716243 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.716269 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:08.716298 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:08.716897 22921 mvcc.cc:205] Tried to move back new op lower bound from 7332982246217646080 to 7332982246007586816. Current Snapshot: MvccSnapshot[applied={T|T < 7332982246061293568}]
W20260924 19:41:08.731886 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.730207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.0.98.61:32853)
0924 19:41:08.730252 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.730255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.731206 (+ 951us) server_negotiation.cc:407] Connection header received
0924 19:41:08.731309 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.731310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.731321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.731350 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.731812 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34576: BlockingRecv error: recv got EOF from 127.0.0.1:34576 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:41:08.732832 22862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.730614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.0.98.60:42257)
0924 19:41:08.730652 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.730654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.732291 (+ 1637us) server_negotiation.cc:407] Connection header received
0924 19:41:08.732391 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.732392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.732403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.732431 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.732767 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40568: BlockingRecv error: recv got EOF from 127.0.0.1:40568 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:41:08.733716 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.730942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.0.98.62:40955)
0924 19:41:08.730981 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.730982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.733185 (+ 2203us) server_negotiation.cc:407] Connection header received
0924 19:41:08.733276 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.733278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.733291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.733319 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.733648 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53494: BlockingRecv error: recv got EOF from 127.0.0.1:53494 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:41:08.741117 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.740249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.98.61:32853)
0924 19:41:08.740293 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.740295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.740539 (+ 244us) server_negotiation.cc:407] Connection header received
0924 19:41:08.740648 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.740649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.740668 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.740697 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.741058 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34592: BlockingRecv error: recv got EOF from 127.0.0.1:34592 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:41:08.744400 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.743527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.0.98.61:32853)
0924 19:41:08.743560 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.743561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.743848 (+ 287us) server_negotiation.cc:407] Connection header received
0924 19:41:08.743981 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.743982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.743994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.744024 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.744347 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34598: BlockingRecv error: recv got EOF from 127.0.0.1:34598 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:41:08.748431 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.747640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.0.98.61:32853)
0924 19:41:08.747674 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.747676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.747901 (+ 225us) server_negotiation.cc:407] Connection header received
0924 19:41:08.748034 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.748035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.748046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.748072 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.748381 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34612: BlockingRecv error: recv got EOF from 127.0.0.1:34612 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:41:08.754485 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.753684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.98.61:32853)
0924 19:41:08.753721 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.753723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.753958 (+ 235us) server_negotiation.cc:407] Connection header received
0924 19:41:08.754055 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.754056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.754066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.754094 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.754418 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34616: BlockingRecv error: recv got EOF from 127.0.0.1:34616 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:41:08.764649 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.763773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.0.98.61:32853)
0924 19:41:08.763814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.763815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.764095 (+ 280us) server_negotiation.cc:407] Connection header received
0924 19:41:08.764191 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.764192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.764203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.764231 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.764589 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34632: BlockingRecv error: recv got EOF from 127.0.0.1:34632 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:41:08.782838 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.781960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.0.98.61:32853)
0924 19:41:08.782005 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.782007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.782273 (+ 266us) server_negotiation.cc:407] Connection header received
0924 19:41:08.782365 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.782367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.782378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.782405 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.782774 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34644: BlockingRecv error: recv got EOF from 127.0.0.1:34644 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:41:08.817195 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.816249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.0.98.61:32853)
0924 19:41:08.816295 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.816297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.816611 (+ 314us) server_negotiation.cc:407] Connection header received
0924 19:41:08.816713 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.816714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.816726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.816757 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.817134 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34648: BlockingRecv error: recv got EOF from 127.0.0.1:34648 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:41:08.883242 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:08.882639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.61:32853)
0924 19:41:08.882704 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:08.882706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:08.882769 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:41:08.882823 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:08.882824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:08.882834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:08.882884 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:08.883178 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34650: BlockingRecv error: recv got EOF from 127.0.0.1:34650 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 19:41:09.013140 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:09.012393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.0.98.61:32853)
0924 19:41:09.012462 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.012465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.012557 (+ 92us) server_negotiation.cc:407] Connection header received
0924 19:41:09.012692 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.012693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.012706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.012750 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:09.013079 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34662: BlockingRecv error: recv got EOF from 127.0.0.1:34662 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:41:09.197319 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef5b5a2b314fd796f93b290fa2a125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } peers { permanent_uuid: "561bd622bb6b46df98331bc50b73318e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32853 } } peers { permanent_uuid: "170101d7e7d64d6cb87014c7bf965b79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42257 } } }
I20260924 19:41:09.197466 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ef5b5a2b314fd796f93b290fa2a125 [sys.catalog]: This master's current role is: LEADER
W20260924 19:41:09.271166 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:09.270503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.0.98.61:32853)
0924 19:41:09.270565 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.270567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.270592 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:41:09.270682 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.270684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.270697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.270750 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:09.271102 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34674: BlockingRecv error: recv got EOF from 127.0.0.1:34674 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:41:09.632345 22875 catalog_manager.cc:1259] Loaded cluster ID: 98f2ba09c2dc459cbe361ae3f230d3bf
I20260924 19:41:09.632421 22875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: loading cluster ID for follower catalog manager: success
I20260924 19:41:09.633222 22875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: acquiring CA information for follower catalog manager: success
I20260924 19:41:09.633505 22875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 170101d7e7d64d6cb87014c7bf965b79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:41:09.714402 22919 catalog_manager.cc:1259] Loaded cluster ID: 98f2ba09c2dc459cbe361ae3f230d3bf
I20260924 19:41:09.714476 22919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: loading cluster ID for follower catalog manager: success
I20260924 19:41:09.715245 22919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: acquiring CA information for follower catalog manager: success
I20260924 19:41:09.715518 22919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 561bd622bb6b46df98331bc50b73318e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:41:09.786450 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:09.784167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.98.61:32853)
0924 19:41:09.784300 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.784302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.784351 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:41:09.784403 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.784404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.784452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.784519 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:09.784634 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.785116 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.785857 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.786048 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.786093 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:09.786200 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:09.786302 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:09.786353 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:09.786369 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 19:41:09.786385 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260924 19:41:09.790174 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:09.788111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.98.60:42257)
0924 19:41:09.788224 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.788226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.788235 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:41:09.788271 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.788272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.788305 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.788366 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:09.788437 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.788915 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.789580 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.789773 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.789814 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:09.789908 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:09.790000 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:09.790050 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:09.790093 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:41:09.790109 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 19:41:09.794440 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:09.791784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.0.98.61:32853)
0924 19:41:09.791817 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.791818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.791828 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:41:09.791852 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.791853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.791863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.791917 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:09.792071 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.792788 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.793794 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.794009 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.794064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:09.794183 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:09.794215 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:09.794258 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:09.794342 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:41:09.794367 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:41:09.794598 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:09.791719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.0.98.60:42257)
0924 19:41:09.791813 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.791815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.792110 (+ 295us) server_negotiation.cc:407] Connection header received
0924 19:41:09.792132 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.792132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.792140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.792178 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:09.792288 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.792730 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.793761 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.794031 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.794092 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:09.794228 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:09.794259 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:09.794304 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:09.794511 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:41:09.794532 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:41:09.794842 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:09.791650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.0.98.62:40955)
0924 19:41:09.791890 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:09.791892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:09.791899 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:41:09.791962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:09.791963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:09.792010 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:09.792082 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:09.792235 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.792958 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.793650 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:09.793935 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:09.793995 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:09.794184 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:09.794283 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:09.794345 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:09.794407 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:41:09.794428 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":53,"threads_started":1}
I20260924 19:41:09.796327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22638
I20260924 19:41:09.800884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22713
I20260924 19:41:09.805070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 22788
2026-09-24T19:41:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2676 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:41:09 dist-test-slave-6kf3 krb5kdc[23010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:41:09 dist-test-slave-6kf3 krb5kdc[23010](info): set up 2 sockets
Sep 24 19:41:09 dist-test-slave-6kf3 krb5kdc[23010](info): commencing operation
krb5kdc: starting...
W20260924 19:41:10.756177 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/test-user.keytab.
Sep 24 19:41:10 dist-test-slave-6kf3 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:41:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:41:10.866685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:41:10.946180 23023 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:10.946403 23023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:10.946419 23023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:10.946437 23023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:10.947530 23023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:10.947561 23023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:10.947574 23023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:10.947585 23023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:10.947595 23023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:10.949025 23023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:10.949218 23023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:10.949404 23023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:41:10.954169 23023 server_base.cc:1034] running on GCE node
W20260924 19:41:10.954164 23029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:10.954159 23031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:10.954207 23028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:10.954627 23023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:10.954896 23023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:10.955996 23023 hybrid_clock.cc:648] HybridClock initialized: now 1790278870955971 us; error 39 us; skew 500 ppm
Sep 24 19:41:10 dist-test-slave-6kf3 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:10.958849 23023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:41:10.959262 23023 webserver.cc:460] Webserver started at http://127.0.98.62:43767/ using document root <none> and password file <none>
I20260924 19:41:10.959467 23023 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:10.959510 23023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:10.959606 23023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:10.960685 23023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data/instance:
uuid: "89c6b674ee69496ba3a65e8fe6ee08d4"
format_stamp: "Formatted at 2026-09-24 19:41:10 on dist-test-slave-6kf3"
server_key: "81e886b9ed0484819c226a0ca5516bf5"
server_key_iv: "c9d72579e41d4805fd3a5055de8cae71"
server_key_version: "encryptionkey@0"
I20260924 19:41:10.960987 23023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance:
uuid: "89c6b674ee69496ba3a65e8fe6ee08d4"
format_stamp: "Formatted at 2026-09-24 19:41:10 on dist-test-slave-6kf3"
server_key: "81e886b9ed0484819c226a0ca5516bf5"
server_key_iv: "c9d72579e41d4805fd3a5055de8cae71"
server_key_version: "encryptionkey@0"
I20260924 19:41:10.962194 23023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:41:10.962849 23038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:10.963004 23023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:10.963091 23023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal
uuid: "89c6b674ee69496ba3a65e8fe6ee08d4"
format_stamp: "Formatted at 2026-09-24 19:41:10 on dist-test-slave-6kf3"
server_key: "81e886b9ed0484819c226a0ca5516bf5"
server_key_iv: "c9d72579e41d4805fd3a5055de8cae71"
server_key_version: "encryptionkey@0"
I20260924 19:41:10.963166 23023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:10.976949 23023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:10.979074 23023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:10.979210 23023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:10.983000 23023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41621
I20260924 19:41:10.983088 23090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41621 every 8 connection(s)
I20260924 19:41:10.983363 23023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:41:10.984436 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:10.985956 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:10.991286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 23023
I20260924 19:41:10.991366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-0/wal/instance
I20260924 19:41:10.991523 392 external_mini_cluster.cc:1428] Setting key abc2ac93c72eaeabb60840268f7b41df
Sep 24 19:41:10 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:10.995160 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:10.986715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.0.98.62:41621)
0924 19:41:10.986849 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:10.986858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:10.986968 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:41:10.987501 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:10.987503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:10.987520 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:10.987591 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:10.988131 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:10.988946 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:10.989784 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:10.990083 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:10.990182 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:10.992132 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:10.992140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:10.992140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:10.992147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:10.993735 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:10.994141 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:10.994144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:10.994144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:10.994219 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:10.994423 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:10.994423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:10.994423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:10.994629 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:10.994685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:10.994894 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:41:10.994994 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 19:41:10.995185 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:10.986664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.0.1:48224)
0924 19:41:10.986840 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:10.986880 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:10.987356 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:10.987608 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:10.987610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:10.987858 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:10.988048 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:10.988053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:10.989039 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:10.989040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:10.989714 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:10.989716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:10.990169 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:10.990170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:10.990230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:10.990662 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:10.990670 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:10.992040 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:10.993814 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:10.993820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:10.993822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:10.994063 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:10.994271 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:10.994271 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:10.994272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:10.994369 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:10.994683 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:10.994689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:10.994765 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 19:41:10.994865 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:41:10.995078 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":54,"thread_start_us":86,"threads_started":1}
I20260924 19:41:10.995949 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:10.996284 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:10.996175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48810)
0924 19:41:10.996222 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:10.996264 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:41:10.996385 23039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
I20260924 19:41:10.996747 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:10.996683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48824)
0924 19:41:10.996710 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:10.996728 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 19:41:10.996815 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37179: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:41:11.014711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37179
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:41:11.046032 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } attempt: 1
I20260924 19:41:11.046545 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.046389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48828)
0924 19:41:11.046454 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.046501 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 19:41:11.046854 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.046779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48834)
0924 19:41:11.046812 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.046836 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:41:11.046945 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37179: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
W20260924 19:41:11.094178 23098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:11.094396 23098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:11.094413 23098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:11.094431 23098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:41:11.095569 23098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:11.095615 23098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:11.095628 23098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:11.095639 23098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:11.095649 23098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:11.097129 23098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:11.097352 23098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:11.097550 23098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:11.102070 23106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:11.102169 23103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:11.102294 23098 server_base.cc:1034] running on GCE node
W20260924 19:41:11.102088 23104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:11.102608 23098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:11.102906 23098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:11.104013 23098 hybrid_clock.cc:648] HybridClock initialized: now 1790278871103993 us; error 27 us; skew 500 ppm
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:11.106421 23098 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:41:11.106812 23098 webserver.cc:460] Webserver started at http://127.0.98.61:35421/ using document root <none> and password file <none>
I20260924 19:41:11.106993 23098 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:11.107030 23098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:11.107118 23098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:11.108158 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data/instance:
uuid: "400ba7cb08f047b0bc1f0e233b2baf9d"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "3f8504ed1435b2f5814930addafb30b9"
server_key_iv: "03319b53afd28138b1b672258792ac89"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.108449 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal/instance:
uuid: "400ba7cb08f047b0bc1f0e233b2baf9d"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "3f8504ed1435b2f5814930addafb30b9"
server_key_iv: "03319b53afd28138b1b672258792ac89"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.109731 23098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:41:11.110437 23113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:11.110570 23098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:41:11.110658 23098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal
uuid: "400ba7cb08f047b0bc1f0e233b2baf9d"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "3f8504ed1435b2f5814930addafb30b9"
server_key_iv: "03319b53afd28138b1b672258792ac89"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.110718 23098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:11.115109 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } attempt: 2
I20260924 19:41:11.115512 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.115386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48848)
0924 19:41:11.115439 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.115482 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:41:11.115797 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.115710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48850)
0924 19:41:11.115757 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.115781 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 19:41:11.115908 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37179: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37179: connect: Connection refused (error 111)
I20260924 19:41:11.126401 23098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:11.128481 23098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:11.128603 23098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:11.132282 23098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37179
I20260924 19:41:11.132318 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37179 every 8 connection(s)
I20260924 19:41:11.132632 23098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:41:11.133558 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:11.135033 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.139272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 23098
I20260924 19:41:11.139377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-1/wal/instance
I20260924 19:41:11.139601 392 external_mini_cluster.cc:1428] Setting key 15af2ec73e1f98dfab631a87f0d11a93
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:11.143859 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.135672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.98.62:41621)
0924 19:41:11.135722 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.135725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.135920 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:41:11.136417 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.136418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.136431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.136481 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.137086 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.137567 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.138491 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.138688 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.138756 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.141180 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.141181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.141182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.141191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.142555 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.142936 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.142938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.142938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.143011 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.143254 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.143254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.143255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.143395 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.143456 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.143683 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.143763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:41:11.143976 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.135646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.0.1:48230)
0924 19:41:11.135799 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.135837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.136319 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.136546 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.136550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.136805 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.136998 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.137002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.137650 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.137651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.138413 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.138414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.138771 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.138773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.139066 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.139603 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.139613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.141085 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.142635 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.142641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.142643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.142868 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.143075 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.143076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.143076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.143196 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.143454 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.143457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.143571 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.143643 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.143784 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260924 19:41:11.144531 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:41:11.152659 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.144755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48860)
0924 19:41:11.144804 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.144816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.144859 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.145076 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.145077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.145212 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.145271 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.145273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.146064 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.146065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.146836 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.146839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.147272 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.147273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.147347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.147654 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.147658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.149991 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.151658 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.151659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.151659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.151845 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.152093 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.152094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.152094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.152150 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.152423 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.152423 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.152457 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.152512 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.152591 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:41:11.152777 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.144816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.98.61:37179)
0924 19:41:11.144952 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.144957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.144973 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:41:11.144991 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.144993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.145004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.145061 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.145333 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.145991 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.146905 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.147207 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.147260 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.150108 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.150115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.150116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.150122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.151574 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.151962 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.151966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.151966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.152025 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.152202 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.152203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.152203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.152371 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.152447 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.152549 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.152653 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 19:41:11.153671 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.154029 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.153887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57832)
0924 19:41:11.153927 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.153969 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:41:11.154166 23115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
I20260924 19:41:11.154572 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.154486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57834)
0924 19:41:11.154533 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.154556 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 19:41:11.154666 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:41:11.164108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm9q4Up/build/release/bin/kudu
/tmp/dist-test-taskm9q4Up/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34311
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:41:11.203908 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } attempt: 1
I20260924 19:41:11.204456 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.204314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57842)
0924 19:41:11.204372 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.204420 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:41:11.204798 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.204702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57848)
0924 19:41:11.204745 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.204775 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:41:11.204910 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
I20260924 19:41:11.207060 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } attempt: 3
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:41:11.214004 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.207363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48864)
0924 19:41:11.207422 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.207433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.207496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.207620 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.207620 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.207758 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.207832 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.207834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.208790 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.208791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.209322 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.209324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.209804 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.209805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.209869 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.210318 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.210323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.211485 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.213091 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.213092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.213093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.213262 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.213449 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.213449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.213449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.213495 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.213762 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.213762 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.213806 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.213859 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.213934 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:41:11.214063 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.207405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.0.98.61:37179)
0924 19:41:11.207447 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.207449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.207478 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:41:11.207543 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.207544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.207554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.207609 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.207891 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.208661 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.209398 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.209730 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.209795 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.211570 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.211571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.211571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.211579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.213007 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.213327 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.213328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.213329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.213391 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.213540 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.213540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.213541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.213697 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.213759 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.213894 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.213975 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:41:11.214696 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.214996 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.214905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57862)
0924 19:41:11.214945 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.214971 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:41:11.215263 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.215188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57866)
0924 19:41:11.215223 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.215244 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:41:11.215341 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
W20260924 19:41:11.245318 23173 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:41:11.245536 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:41:11.245559 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:41:11.245589 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:41:11.246507 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } attempt: 1
W20260924 19:41:11.246745 23173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:41:11.246783 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:41:11.246801 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:41:11.246819 23173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:41:11.246842 23173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:41:11.246963 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.246825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57882)
0924 19:41:11.246880 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.246926 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:41:11.247347 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.247245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57884)
0924 19:41:11.247293 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.247324 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:41:11.247454 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
I20260924 19:41:11.248551 23173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41621,127.0.98.61:37179,127.0.98.60:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:41:11.248783 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:41:11.248986 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:41:11.253413 23178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:11.253489 23181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:41:11.253548 23179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:41:11.253759 23173 server_base.cc:1034] running on GCE node
I20260924 19:41:11.253991 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:41:11.254295 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:41:11.255445 23173 hybrid_clock.cc:648] HybridClock initialized: now 1790278871255419 us; error 34 us; skew 500 ppm
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:41:11.257869 23173 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:41:11.258324 23173 webserver.cc:460] Webserver started at http://127.0.98.60:46555/ using document root <none> and password file <none>
I20260924 19:41:11.258526 23173 fs_manager.cc:359] Metadata directory not provided
I20260924 19:41:11.258575 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:41:11.258695 23173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:41:11.259716 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data/instance:
uuid: "7d3074de7ce141118462a710363737ee"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "ec7b46b80a3a84e9c73eb3e96c492d55"
server_key_iv: "b5d1d0354f46fc49e8fc5c25d4fa90f5"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.260076 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal/instance:
uuid: "7d3074de7ce141118462a710363737ee"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "ec7b46b80a3a84e9c73eb3e96c492d55"
server_key_iv: "b5d1d0354f46fc49e8fc5c25d4fa90f5"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.261390 23173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:41:11.262125 23188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:41:11.262279 23173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:41:11.262364 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal
uuid: "7d3074de7ce141118462a710363737ee"
format_stamp: "Formatted at 2026-09-24 19:41:11 on dist-test-slave-6kf3"
server_key: "ec7b46b80a3a84e9c73eb3e96c492d55"
server_key_iv: "b5d1d0354f46fc49e8fc5c25d4fa90f5"
server_key_version: "encryptionkey@0"
I20260924 19:41:11.262424 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:41:11.273080 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } attempt: 2
I20260924 19:41:11.273558 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.273410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57900)
0924 19:41:11.273474 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.273516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:41:11.273939 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.273837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57902)
0924 19:41:11.273880 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.273912 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:41:11.274034 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34311: connect: Connection refused (error 111)
I20260924 19:41:11.276216 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:41:11.278228 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:41:11.278332 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:41:11.282023 23173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34311
I20260924 19:41:11.282061 23240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34311 every 8 connection(s)
I20260924 19:41:11.282377 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:41:11.283310 23241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:41:11.284893 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.288461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm9q4Up/build/release/bin/kudu as pid 23173
I20260924 19:41:11.288548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm9q4Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278584646480-392-0/minicluster-data/master-2/wal/instance
I20260924 19:41:11.288738 392 external_mini_cluster.cc:1428] Setting key c6516c922010aec3ed1499c34663077f
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:11.294570 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.285587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.0.98.62:41621)
0924 19:41:11.285681 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.285684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.285838 (+ 154us) server_negotiation.cc:407] Connection header received
0924 19:41:11.286356 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.286357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.286370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.286434 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.287051 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.287530 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.288619 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.288895 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.288954 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.291209 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.291210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.291210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.291219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.293246 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.293653 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.293655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.293655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.293712 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.293972 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.293973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.293973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.294068 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.294110 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.294429 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.294490 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:41:11.294791 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.285502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.0.1:48244)
0924 19:41:11.285714 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.285756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.286250 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.286509 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.286512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.286757 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.286956 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.286961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.287618 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.287620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.288508 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.288511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.289000 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.289002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.289185 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.289744 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.289752 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.291127 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.293331 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.293337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.293339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.293580 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.293771 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.293772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.293773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.293913 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.294121 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.294127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.294284 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.294394 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.294564 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260924 19:41:11.295521 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.297470 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.289167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.98.60:34311)
0924 19:41:11.289303 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.289308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.289353 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:41:11.289385 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.289387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.289409 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.289487 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.290099 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.290871 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.291736 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.292193 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.292252 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.294740 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.294748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.294748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.294755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.296403 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.296749 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.296751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.296751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.296796 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.296943 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.296944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.296944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.297110 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.297234 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.297298 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.297369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:41:11.302688 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.295772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37179 (local address 127.0.0.1:48866)
0924 19:41:11.295821 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.295831 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.295874 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.296029 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.296030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.296191 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.296249 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.296251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.296997 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.296998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.297687 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.297690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.297997 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.297998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.298072 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.298439 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.298444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.299593 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.301211 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.301212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.301212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.301376 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.302017 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.302019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.302020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.302090 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.302403 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.302403 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.302436 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.302489 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.302599 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":18}
I20260924 19:41:11.302686 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.295852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.61:37179)
0924 19:41:11.295906 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.295908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.295914 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:41:11.295954 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.295955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.295966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.296024 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.296310 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.296936 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.297760 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.297946 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.298000 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.299671 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.299673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.299673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.299680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.301116 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.301445 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.301447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.301447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.301499 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.302177 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.302178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.302178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.302321 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.302378 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.302525 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.302590 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:41:11.303308 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:41:11.304926 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.298240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.98.62:41621)
0924 19:41:11.298289 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.298291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.298297 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:41:11.298316 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.298317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.298326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.298371 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.298909 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.299348 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.300158 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.300348 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.300416 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.302375 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.302377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.302377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.302383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.304004 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.304272 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.304273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.304273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.304336 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.304477 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.304478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.304478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.304601 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.304771 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.304804 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.304864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:41:11.310066 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.303540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57926)
0924 19:41:11.303626 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.303634 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.303662 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.303784 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.303785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.303882 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.303910 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.303911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.304686 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.304688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.305349 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.305352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.305736 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.305737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.305781 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.306146 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.306151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.307213 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.308566 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.308568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.308568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.308777 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.309436 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.309437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.309437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.309514 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.309751 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.309751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.309796 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.309868 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.309946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 19:41:11.310117 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.303599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.98.60:34311)
0924 19:41:11.303643 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.303645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.303657 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:41:11.303701 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.303702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.303711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.303783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.303993 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.304614 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.305416 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.305671 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.305743 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.307283 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.307284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.307285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.307292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.308486 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.308858 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.308858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.308859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.308900 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.309592 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.309592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.309593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.309689 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.309737 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.309897 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.309994 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":25}
I20260924 19:41:11.311888 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: Bootstrap starting.
I20260924 19:41:11.312235 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.305559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.0.98.61:37179)
0924 19:41:11.305600 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.305602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.305619 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:41:11.305636 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.305637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.305644 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.305683 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.306077 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.306502 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.307128 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.307356 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.307419 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.309810 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.309811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.309811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.309816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.311181 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.311543 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.311544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.311544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.311585 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.311763 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.311764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.311764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.311847 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.311968 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.312120 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.312177 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:41:11.312923 23241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:11.313253 23241 log.cc:824] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:11.313710 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:41:11.314076 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: No bootstrap required, opened a new log
I20260924 19:41:11.314570 23065 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250
I20260924 19:41:11.316140 23241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.316299 23241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3074de7ce141118462a710363737ee, State: Initialized, Role: FOLLOWER
I20260924 19:41:11.316385 23241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.316705 23241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:11.316675 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.316756 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:41:11.318550 23258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:41:11.318608 23258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:41:11.318810 23173 master_runner.cc:186] Error getting master registration for 127.0.98.62:41621: OK, instance_id { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" instance_seqno: 1790278870981945 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:41:11.321720 23173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:41:11.322593 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } attempt: 2
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:11.329916 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.322998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.0.98.60:34311)
0924 19:41:11.323044 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.323046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.323055 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:41:11.323079 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.323079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.323090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.323146 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.323500 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.324025 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.324943 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.325238 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.325302 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.327439 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.327440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.327441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.327449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.328844 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.329165 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.329166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.329167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.329225 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.329398 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.329399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.329399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.329540 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.329604 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.329748 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.329817 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:41:11.329916 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.322875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57938)
0924 19:41:11.322934 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.322949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.322986 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.323168 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.323169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.323346 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.323399 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.323401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.324098 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.324099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.324869 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.324872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.325306 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.325307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.325365 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.325834 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.325839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.327336 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.328925 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.328927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.328927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.329100 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.329282 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.329283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.329284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.329344 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.329603 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.329604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.329653 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.329719 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.329817 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:41:11.331224 23091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: Bootstrap starting.
I20260924 19:41:11.331871 23091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:11.332175 23091 log.cc:824] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:11.333051 23091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: No bootstrap required, opened a new log
I20260924 19:41:11.334462 23091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.334610 23091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c6b674ee69496ba3a65e8fe6ee08d4, State: Initialized, Role: FOLLOWER
I20260924 19:41:11.334692 23091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.335042 23091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:11.334990 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.335107 23065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:41:11.335186 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.335219 23065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:11.336156 23065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.337252 23215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d3074de7ce141118462a710363737ee"
I20260924 19:41:11.337396 23215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:11.338287 23215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c6b674ee69496ba3a65e8fe6ee08d4 in term 1.
I20260924 19:41:11.336939 23065 leader_election.cc:288] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [CANDIDATE]: Term 1 election: Requested vote from peers 400ba7cb08f047b0bc1f0e233b2baf9d (127.0.98.61:37179), 7d3074de7ce141118462a710363737ee (127.0.98.60:34311)
I20260924 19:41:11.338549 23040 leader_election.cc:302] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3074de7ce141118462a710363737ee, 89c6b674ee69496ba3a65e8fe6ee08d4; no voters:
I20260924 19:41:11.338688 23261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:41:11.338876 23261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [term 1 LEADER]: Becoming Leader. State: Replica: 89c6b674ee69496ba3a65e8fe6ee08d4, State: Running, Role: LEADER
I20260924 19:41:11.338976 23261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.339392 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c6b674ee69496ba3a65e8fe6ee08d4. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.339483 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: This master's current role is: LEADER
W20260924 19:41:11.337428 23272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:41:11.339695 23272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:41:11.339193 23023 master_runner.cc:186] Error getting master registration for 127.0.98.61:37179: OK, instance_id { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" instance_seqno: 1790278871131214 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:41:11.341511 23023 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:41:11.341958 23280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:41:11.342160 23280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:41:11.344558 23215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [term 1 FOLLOWER]: Refusing update from remote peer 89c6b674ee69496ba3a65e8fe6ee08d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:41:11.345009 23261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:41:11.347393 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c6b674ee69496ba3a65e8fe6ee08d4. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.347571 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.349802 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.339973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.98.62:41621)
0924 19:41:11.340014 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.340016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.340250 (+ 234us) server_negotiation.cc:407] Connection header received
0924 19:41:11.340428 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.340429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.340441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.340475 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.341371 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.341966 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.343006 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.343269 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.343326 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.345761 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.345763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.345763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.345771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.347850 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.348682 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.348683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.348684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.348752 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.349094 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.349109 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.349109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.349252 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.349344 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.349638 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.349714 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:41:11.352728 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.341781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.0.98.61:37179)
0924 19:41:11.341833 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.341835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.342922 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 19:41:11.343016 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.343017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.343037 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.343066 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.343755 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.344444 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.346647 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.346923 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.347012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.349859 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.349861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.349861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.349867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.351483 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.351971 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.351972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.351972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.352022 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.352212 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.352213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.352214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.352335 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.352430 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.352551 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.352635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:41:11.353701 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.344317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.0.98.60:34311)
0924 19:41:11.344718 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.344720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.344728 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:41:11.344745 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.344746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.344754 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.344804 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.346888 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.347597 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.348624 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.348904 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.348940 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.350584 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.350586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.350586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.350593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.352372 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.352880 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.352881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.352881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.352938 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.353173 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.353174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.353174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.353308 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.353392 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.353543 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.353613 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260924 19:41:11.357002 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.357098 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:41:11.357443 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.357527 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.357712 23283 mvcc.cc:205] Tried to move back new op lower bound from 7332982257021952000 to 7332982257005195264. Current Snapshot: MvccSnapshot[applied={T|T < 7332982257021952000 or (T in {7332982257021952000})}]
I20260924 19:41:11.357935 23280 catalog_manager.cc:1347] Generated new cluster ID: afccd6542f654ad88973585b6297a7ef
I20260924 19:41:11.357995 23280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:41:11.365171 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } attempt: 3
I20260924 19:41:11.366822 23280 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:41:11.367558 23280 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 19:41:11 dist-test-slave-6kf3 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:41:11.373075 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.365490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34311 (local address 127.0.0.1:57946)
0924 19:41:11.365576 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 19:41:11.365587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:41:11.365622 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:41:11.365878 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:41:11.365879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:41:11.366074 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:41:11.366123 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.366126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.367023 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.367025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.367749 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:41:11.367752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.368201 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.368202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:41:11.368258 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.368738 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:41:11.368743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:41:11.370054 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:41:11.372059 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.372060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.372061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.372263 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.372450 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:41:11.372451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:41:11.372451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:41:11.372527 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:41:11.372773 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:41:11.372774 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:41:11.372816 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:41:11.372875 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:41:11.372971 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:41:11.373625 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:11.365706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.98.60:34311)
0924 19:41:11.365746 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.365748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.365755 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:41:11.365775 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.365776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.365785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.365829 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.366218 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.366933 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.367819 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:11.368134 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:11.368190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:11.370137 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:41:11.370139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:41:11.370139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:41:11.370145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:41:11.371978 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.372325 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.372326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.372326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.372396 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:41:11.372594 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:41:11.372595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:41:11.372596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:41:11.372718 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:41:11.372771 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:11.373458 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 19:41:11.373540 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:41:11.375057 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: Bootstrap starting.
I20260924 19:41:11.375773 23166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: Neither blocks nor log segments found. Creating new log.
I20260924 19:41:11.376080 23166 log.cc:824] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:41:11.376969 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: No bootstrap required, opened a new log
I20260924 19:41:11.379009 23166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.379142 23166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400ba7cb08f047b0bc1f0e233b2baf9d, State: Initialized, Role: FOLLOWER
I20260924 19:41:11.379230 23166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } }
I20260924 19:41:11.379534 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.379611 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:41:11.379746 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.380441 23139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [term 1 FOLLOWER]: Refusing update from remote peer 89c6b674ee69496ba3a65e8fe6ee08d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:41:11.379590 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d"
I20260924 19:41:11.380738 23261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:41:11.381222 23140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c6b674ee69496ba3a65e8fe6ee08d4 in term 1.
I20260924 19:41:11.382851 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c6b674ee69496ba3a65e8fe6ee08d4. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.382930 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.382967 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.383023 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:41:11.379562 23166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:41:11.383560 23290 mvcc.cc:205] Tried to move back new op lower bound from 7332982257111736320 to 7332982257005195264. Current Snapshot: MvccSnapshot[applied={T|T < 7332982257021952000}]
I20260924 19:41:11.383726 23285 mvcc.cc:205] Tried to move back new op lower bound from 7332982257021952000 to 7332982257005195264. Current Snapshot: MvccSnapshot[applied={T|T < 7332982257021952000}]
I20260924 19:41:11.385035 23307 catalog_manager.cc:1259] Loaded cluster ID: afccd6542f654ad88973585b6297a7ef
I20260924 19:41:11.385076 23307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: loading cluster ID for follower catalog manager: success
I20260924 19:41:11.385474 23280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4: Generated new TSK 0
I20260924 19:41:11.385664 23280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:41:11.386708 23307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: acquiring CA information for follower catalog manager: success
W20260924 19:41:11.386817 23307 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260924 19:41:11.388698 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6b674ee69496ba3a65e8fe6ee08d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } peers { permanent_uuid: "400ba7cb08f047b0bc1f0e233b2baf9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37179 } } peers { permanent_uuid: "7d3074de7ce141118462a710363737ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34311 } } }
I20260924 19:41:11.388794 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6b674ee69496ba3a65e8fe6ee08d4 [sys.catalog]: This master's current role is: LEADER
W20260924 19:41:11.424705 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.416289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.98.61:37179)
0924 19:41:11.416338 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.416340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.423981 (+ 7641us) server_negotiation.cc:407] Connection header received
0924 19:41:11.424103 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.424104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.424116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.424147 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.424628 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48888: BlockingRecv error: recv got EOF from 127.0.0.1:48888 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 19:41:11.425454 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.423525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.0.98.60:34311)
0924 19:41:11.423563 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.423564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.424953 (+ 1389us) server_negotiation.cc:407] Connection header received
0924 19:41:11.425060 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.425061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.425071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.425099 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.425391 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57956: BlockingRecv error: recv got EOF from 127.0.0.1:57956 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:41:11.426138 23094 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.423316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.98.62:41621)
0924 19:41:11.423362 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.423364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.425662 (+ 2298us) server_negotiation.cc:407] Connection header received
0924 19:41:11.425753 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.425754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.425764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.425791 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.426074 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48270: BlockingRecv error: recv got EOF from 127.0.0.1:48270 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 19:41:11.428537 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.427743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.0.98.61:37179)
0924 19:41:11.427778 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.427780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.428045 (+ 265us) server_negotiation.cc:407] Connection header received
0924 19:41:11.428151 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.428152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.428171 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.428198 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.428488 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48902: BlockingRecv error: recv got EOF from 127.0.0.1:48902 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:41:11.431449 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.430748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.0.98.61:37179)
0924 19:41:11.430780 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.430782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.430983 (+ 201us) server_negotiation.cc:407] Connection header received
0924 19:41:11.431085 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.431086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.431096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.431121 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.431403 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48914: BlockingRecv error: recv got EOF from 127.0.0.1:48914 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:41:11.435292 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.434505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.0.98.61:37179)
0924 19:41:11.434537 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.434539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.434806 (+ 267us) server_negotiation.cc:407] Connection header received
0924 19:41:11.434895 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.434895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.434905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.434931 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.435246 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48924: BlockingRecv error: recv got EOF from 127.0.0.1:48924 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:41:11.441293 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.440496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.98.61:37179)
0924 19:41:11.440533 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.440535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.440798 (+ 263us) server_negotiation.cc:407] Connection header received
0924 19:41:11.440890 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.440891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.440901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.440927 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.441250 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:41:11.451267 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.450461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.0.98.61:37179)
0924 19:41:11.450498 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.450500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.450766 (+ 266us) server_negotiation.cc:407] Connection header received
0924 19:41:11.450851 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.450852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.450862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.450888 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.451214 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48942: BlockingRecv error: recv got EOF from 127.0.0.1:48942 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:41:11.469558 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.468633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.98.61:37179)
0924 19:41:11.468680 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.468682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.468938 (+ 256us) server_negotiation.cc:407] Connection header received
0924 19:41:11.469060 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.469062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.469073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.469103 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.469497 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48948: BlockingRecv error: recv got EOF from 127.0.0.1:48948 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:41:11.504019 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.503031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.98.61:37179)
0924 19:41:11.503077 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.503079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.503362 (+ 283us) server_negotiation.cc:407] Connection header received
0924 19:41:11.503467 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.503469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.503480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.503511 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.503904 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48964: BlockingRecv error: recv got EOF from 127.0.0.1:48964 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:41:11.570051 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.569465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.0.98.61:37179)
0924 19:41:11.569531 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.569533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.569602 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:41:11.569633 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.569634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.569645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.569705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.570001 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48976: BlockingRecv error: recv got EOF from 127.0.0.1:48976 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 19:41:11.699870 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.699332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.0.98.61:37179)
0924 19:41:11.699412 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.699414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.699450 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:41:11.699476 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.699477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.699487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.699531 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.699810 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48980: BlockingRecv error: recv got EOF from 127.0.0.1:48980 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 19:41:11.957738 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:41:11.957137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.0.98.61:37179)
0924 19:41:11.957208 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:11.957210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:11.957221 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:41:11.957291 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:11.957292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:11.957303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:11.957345 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:41:11.957677 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:41:12.319015 23258 catalog_manager.cc:1259] Loaded cluster ID: afccd6542f654ad88973585b6297a7ef
I20260924 19:41:12.319095 23258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: loading cluster ID for follower catalog manager: success
I20260924 19:41:12.319878 23258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: acquiring CA information for follower catalog manager: success
I20260924 19:41:12.320186 23258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d3074de7ce141118462a710363737ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:41:12.387341 23307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 400ba7cb08f047b0bc1f0e233b2baf9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:41:12.473346 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:12.470838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.61:37179)
0924 19:41:12.470949 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:12.470951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:12.470993 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:41:12.471066 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:12.471067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:12.471125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:12.471197 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:12.471337 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.471836 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.472599 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.472810 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.472866 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:12.473016 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:12.473127 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:12.473194 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:12.473240 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 19:41:12.473260 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260924 19:41:12.477380 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:12.475089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.98.60:34311)
0924 19:41:12.475216 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:12.475218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:12.475224 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:41:12.475257 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:12.475258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:12.475291 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:12.475355 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:12.475476 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.475983 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.476679 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.476866 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.476917 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:12.477106 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:12.477206 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:12.477259 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:12.477288 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:41:12.477303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260924 19:41:12.482084 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:12.479215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.98.62:41621)
0924 19:41:12.479326 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:12.479328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:12.479335 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:41:12.479371 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:12.479372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:12.479421 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:12.479513 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:12.479656 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.480484 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481258 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.481530 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481589 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:12.481767 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:12.481865 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:12.481909 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:12.481968 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:41:12.481998 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260924 19:41:12.482437 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:12.479190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.0.98.60:34311)
0924 19:41:12.479235 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:12.479236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:12.479495 (+ 259us) server_negotiation.cc:407] Connection header received
0924 19:41:12.479524 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:12.479525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:12.479536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:12.479585 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:12.479704 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.480438 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481451 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.481662 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481731 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:12.482185 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:12.482216 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:12.482261 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:12.482322 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:41:12.482341 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:41:12.482916 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:41:12.479101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.0.98.61:37179)
0924 19:41:12.479149 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:41:12.479150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:41:12.479378 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:41:12.479425 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:41:12.479426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:41:12.479435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:41:12.479495 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:41:12.479765 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.480306 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481393 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:41:12.481663 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:41:12.481720 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:41:12.482069 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:41:12.482105 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:41:12.482156 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:41:12.482836 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 19:41:12.482857 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:41:12.483965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 23023
I20260924 19:41:12.488713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 23098
I20260924 19:41:12.492791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm9q4Up/build/release/bin/kudu with pid 23173
2026-09-24T19:41:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2687 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5364 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287849 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS